Typical thing: You try and search for over an hour, then you ask here and 10 minutes later you find the solution yourself:
The problem was with how I referenced the system.drawing.dll in the compileroptions. I did use an absolute path C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\System.Drawing.dll
and that gave me 19 exceptions in some code I did not even write myself.
What worked was to use a relative path. e.g.
PM.ReferencedAssemblies.Add("System.Drawing.dll")
I don't know what the real difference is, but at least this works.