Python GIL, C extensions, and increasing through put (Python 3.2)

Question

I have a server application that accepts requests, needs to run some Python, some of which calls into my C-based extension DLLs. If I completely lock the GIL during the entire request, everything works fine. But I'd like to release it while in the C-based extension DLL when long-ish operations are taking place so more requests can be processed. This is my approach so far:

A. Request arrives on thread I don't control
B.   Call PyGILState_Ensure
C.      Call into Python runtime via PyRun_String  
D.          Python calls my MethodA in my C-based code
E.              Call PyGILState_Release 
F.                  Do long-ish processing
G.              Call PyGILState_Ensure to lock GIL again
H.          Return control to Python for further script processing
I.      Python runtime returns
J.   Call PyGILState_Release

All of the steps above are a single thread (and there are many similar threads in the system).

The call sequence above always throws an exception somewhere in step (H) deep in Python (which I can't seem to figure out from the stack trace). However, if I make an extra call to PyGILState_Ensure at (B) so that the calls at (E) and (F) don't end up doing anything, everything runs perfectly.

Can someone please help me understand what I'm doing wrong? I'd really like to release the GIL during lengthy operations so other requests can make progress.

Answer 1

First, you have to match the PyGILState_Ensure/PyGILState_Release pairs at B and E, and G and J, rather than the more naively-obvious matching matching of B and J and E and G. (If you try it that way, step E releases an uninitialized state, corrupting the interpreter's internal state information.)

But you're using a single state, which doesn't have that problem. (I'm not actually sure this is legal, but I am pretty sure it's safe with every version of CPython, and there's definitely no way it could get things out of order.)

---

Unfortunately, this has a different problem. As the PyGILState_Release docs say, it:

Release any resources previously acquired. After this call, Python’s state will be the same as it was prior to the corresponding PyGILState_Ensure() call…

In other words, you can't carry any interpreter resources across a Release and subsequent Ensure (unless they're protected by an outer Ensure, of course, but in that case you're not actually releasing anything).

So, as soon as you Release the state that was ensured for PyRun_String, the rest of that PyRun_String call becomes invalid. Switching to a newly-acquired state in the middle doesn't help.

---

But I think you're abusing Ensure/Release unnecessarily in the first place. You don't need to register the thread with the interpreter, unregister it, register it again, and unregister it again; all you need to do is release and reacquire the GIL from within the same thread. That's what Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS are for.

As the documentation for PyGILState_Ensure says:

In general, other thread-related APIs may be used between PyGILState_Ensure() and PyGILState_Release() calls as long as the thread state is restored to its previous state before the Release(). For example, normal usage of the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros is acceptable.

So:

PyGILState_STATE state;

B. state = PyGILState_Ensure();
    ...
E.              Py_BEGIN_ALLOW_THREADS
F.                  Do long-ish processing
G.              Py_END_ALLOW_THREADS
    ...
J.  PyGILState_Release(state);