By default B4 pin is analog. Configure it as digital by clearing the ANSB register, bit4
NOTE: Although clearing the bit DID NOT fix the problem. Moving to another pin (with less features) did. So I (fossum) made the assumption that this was at least on some level the correct answer.