Well as it turns out it was me (as suspected). In my code I have numerous try/catches and in the catch clause I have the following
#if DEBUG
System.Diagnostics.Debugger.Break();
#endif
I accidently deployed a version of my code that had the DEBUG switch turned on so it was hitting that code whenever any error occurred and fired up the JIT debugger on the server. Duh. Silly me.