Dunno what the policy is around here for answering your own question, but in case someone stumbles upon this with the same problem:
I was transforming my light position by the model matrix. This doesn't make any sense, because the light position is already in world-coordinates, so there's no need to transform it at all.