Stopping embedded Python

Question

I'm embedding Python interpreter to a C program. However, it might happen that while running some python script via PyRun_SimpleString() will run into infinite loop or execute for too long. Consider PyRun_SimpleString("while 1: pass"); In preventing the main program to block I thought I could run the interpreter in a thread.

How do I stop executing the python script in embedded interpreter running in a thread without killing the whole process?

Is it possible to pass an exception to the interpreter? Should I wrap the script under some other script which would listen to signals?

PS: I could run the python in a separate process but this is not what I want - unless it is the last resort...

---

Update:

So, it works now. Thank you Denis Otkidach, once again!

If I see this right, you have to do two things: tell the interpreter to stop and return -1 in the same thread as your PyRun_SimpleString() is running.

To stop, one has a few possibilities: PyErr_SetString(PyExc_KeyboardInterrupt, "...") or PyErr_SetInterrupt() - the first one might leave Python running a few more instructions and then it stops, the later one stops the execution immediately.

To return -1 you use Py_AddPendingCall() to inject a function call into Python execution. The docs are mentioning it since version 2.7 and 3.1 but it runs on earlier Pythons as well (2.6 here). From 2.7 and 3.1 it should also be thread-safe, meaning you can call it without acquiring GIL (?).

So one could rewrite the example bellow:

int quit() {
    PyErr_SetInterrupt();
    return -1;
}

Answer 1

You can use Py_AddPendingCall() to add a function raising exception to be called on next check interval (see docs on sys.setcheckinterval() for more info). Here is an example with Py_Exit() call (which does works for me, but probably is not what you need), replace it with Py_Finalize() or one of PyErr_Set*():

int quit(void *) {
    Py_Exit(0);
}


PyGILState_STATE state = PyGILState_Ensure();
Py_AddPendingCall(&quit, NULL);
PyGILState_Release(state);

This should be enough for any pure-python code. But note, that some C functions can run for a while as a single operation (there was an example with long running regexp search, but I'm not sure it's still relevant).

Answer 2

Well the python interpreter would have to be running in a separate thread from the embedding program or you will simply never get a chance to interrupt the interpreter.

When you have that, perhaps you can use one of the Python API exception calls to trigger an exception in the interpreter? See Python C API Exceptions

Answer 3

I wanted to be able to interrupt any actively running Python code block, including cancelling an infinite loop or just some long-running code. In my application, single-threaded Python code is submitted and evaluated. An interrupt request can come at any time. This question and answer were crucial in helping me actually achieve that.

Looking at this question in 2019, the solution here did not work for me; I tried to use Py_AddPendingCall(&quit, NULL); in quite a few different ways but the called function never executed as expected, or at all. This caused the Python engine to hang and eventually crash when I submitted Py_FinalizeEx(). I finally found what I needed in this very useful answer to a similar question.

After I start the Python engine, I retrieve the thread id using the threading package in Python code. I parse that into a c long value and save it.

The interrupt function is actually quite simple:
PyGILState_Ensure()
PyThreadState_SetAsyncExc(threadId, PyExc_Exception) using the thread ID that I save from earlier and a generic exception type
PyGILState_Release()