Turns out the problem is a combination of Adobe's Protected Mode and trying to load a .Net 4 DLL through COM interop, but only on Windows XP.
Bizarrely... change any of those variables, and everything works fine. .Net 3.5 and 2.0 both load fine with Protected Mode on. Windows 7 loads the .Net 4 dll fine with Protected Mode on.
Turning off Protected Mode in Adobe for Windows XP fixes the problem so the DLL loads fine.
I hope this helps someone else who runs into this.