Problems with IronPython memory leaks in web application

Question

Has anyone successfully gotten IronPython running without issues in a web environment? I am running into some issues.

First issue, I am not actually running any IronPython-specific scripts, I am implementing the Pygments library so I can get server-side syntax highlighting. The library is about 20+ files.

Besides the fact that the latest IronPython release cannot compile the scripts into a DLL (due to this issue), I do have it running successfully by just copying all the files and dependencies to my bin folder.

The problem is, I went to look at how my w3wp.exe process is doing when doing highlighting and noticed some showstopper issues:

• Even in a totally basic, empty web site under Cassini, highlighting the code select * from table using the SQL lexer causes a 10MB jump every time I execute it (page refresh)... I am explicitly shutting down the engine and using LightweightScopes in a single function call. It starts at about 30MB and with about 20 refreshes it's at 150MB or so.

• In my actual web application, using the SQL lexer (same code) causes my app pool to increase about 200MB/sec (literally, I kill it when it reaches about 1GB) until it either crashes w3wp or slows my PC down to a crawl. This doesn't happen in the empty test site and there are no problems at all in a console app with the same exact code. Other lexers like C# do not cause the huge memory leak but do have the same effect of increasing the memory each time you call the function.

This leads me to believe this a web-specific issue, considering the console app doesn't have any problems (instantiating the runtime does cause a 20MB increase in memory, though).

I am using the 2.7 IPY release and 1.4 release of Pygments.

I don't have the exact code with me at the moment but it looks something like:

var options = something;
options["LightweightScopes"] = ScriptRuntimeHelpers.True; // from another SO post, 'true' didn't seem to work
var engine = Python.CreateEngine(options);
//
// set up search paths here...
//
dynamic scope = whatever;
ScriptSource source = engine.CreateScriptSourceFromFile("myscript.py");

// Execute? Compile? It populates the scope at this point...
source.Compile(scope);

// execute (code, lexer name, style)
// this is a python function I have that calls the Pygments code
var highlighted = scope.generate_html("select * from table", "sql", "monokai");

engine.Shutdown();
return highlighted;

Like I said, I copied this same code in: a) a console app, b) a brand new empty web app, c) my original web app. The console app did not leak memory, but the web apps do.

I also executed the function with both the native Python (python myscript.py) and IPY (ipy myscript.py) and neither had any memory leaks.

Is there some best practice I am missing to dispose of the runtime? I plan on running this in a shared environment, so the possible workaround of instantiating the engine in a different app pool probably won't work for me (also, that huge leak with the 200MB/s is kind of a showstopper).

Currently, unless someone has a miracle cure, I plan to scrap my code and just go with a Javascript syntax highlighter. It's real unfortunate because Pygments is awesome...

Answer 1

That's quite a long story. (TLDR, sorry). This being .NET, you're most likely looking at heap fragmentation rather than memory exhaustion.

If you are really leaking memory then you are holding on to references too long. Review you IDisposables (especially list comprehension can hurt, especially with Linqy things like orderby and distinct tacked on).

I once made an application run 2x as fast, when it was reliably failing by exhausting/fragmenting the heap by tacking a simple .ToList() on the largest enumerations I had to drive a big process.

There are good memory profilers for .NET, but I currently know how to operate only mono --profiler (bad memory). A simple google will help you find a profiler you can use on your environment: let it tell you exactly what objects allocated where are fragmenting your heap

---

PS. since I saw you pointing the finger on Web applications on second scanning of your question, I would add: check what references (indirectly) are being held from Session state (application, session).