Thanks to ratchet freak and Andon M. Coleman.
Ratchet freak pointed out that it was probably a rounding error in the angle at which the light hits the surface (leading to a low dot product between normal and direction).
The cause was that the internal format of the normal direction texture was set to GL_RGBA16F - 16 bit floats which caused the rounding. Setting the format to GL_RGBA32F fixed the rounding by holding 32 bit floats.
Andon M. Coleman pointed out that I should stop using GL_RGBA as an internal format because it is ambiguous and that GL_RGBA8 was appropriate