https://stackoverflow.com/questions/20799422
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
RussianIf MyClass::some_later_event() is being invoked from a C++ thread that is not explicitly managing the Global Interpreter Lock (GIL), then that can result in undefined behavior.
Python and C++ threads.
Lets consider the case where C++ threads are interacting with Python. For example, a C++ thread can be set to invoke the MyClass's signal after a period of time via MyClass.event_in(seconds, value).
This example can become fairly involved, so lets start with the basics: Python's GIL. In short, the GIL is a mutex around the interpreter. If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. In the GDB traceback, the Boost.Signals2 library was likely trying to invoke a Python object without the GIL, resulting in the crash. While managing the GIL is fairly straightforward, it can become complex rather quickly.
First, the module needs to have Python initialize the GIL for threading.
BOOST_PYTHON_MODULE(example)
{
PyEval_InitThreads(); // Initialize GIL to support non-python threads.
// ...
}
For convenience, lets create a simple class to help manage the GIL via scope:
/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
gil_lock() { state_ = PyGILState_Ensure(); }
~gil_lock() { PyGILState_Release(state_); }
private:
PyGILState_STATE state_;
};
Lets identify when the C++ thread will need the GIL:
boost::signals2::signalcan make additional copies of connected objects, as is done when the signal is concurrently invoked.- Invoking a Python objected connected through
boost::signals2::signal. The callback will certainly affect python objects. For example, theselfargument provided to__call__method will increase and decrease an object's reference count.
The MyClass class.
Here is a basic mockup class based on the original code:
/// @brief Mockup class.
class MyClass
{
public:
/// @brief Connect a slot to the signal.
template <typename Slot>
void connect_slot(const Slot& slot)
{
signal_.connect(slot);
}
/// @brief Send an event to the signal.
void event(int value)
{
signal_(value);
}
private:
boost::signals2::signal<void(int)> signal_;
};
As a C++ thread may be invoking MyClass's signal, the lifetime of MyClass must be at least as long as the thread. A good candidate to accomplish this is by having Boost.Python manage MyClass with a boost::shared_ptr.
BOOST_PYTHON_MODULE(example)
{
PyEval_InitThreads(); // Initialize GIL to support non-python threads.
namespace python = boost::python;
python::class_<MyClass, boost::shared_ptr<MyClass>,
boost::noncopyable>("MyClass")
.def("event", &MyClass::event)
// ...
;
}
boost::signals2::signal interacting with python objects.
boost::signals2::signal may make copies when it is invoked. Additionally, there may be C++ slots connected to the signal, so it would be ideal to only lock the GIL when Python slots are being invoked. However, signal does not provide hooks to allow us to acquire the GIL before creating copies of slots or invoking the slot.
In order to avoid having signal create copies of boost::python::object slots, one can use a wrapper class that creates a copy of boost::python::object so that reference counts remain accurate, and manages the copy via shared_ptr. This allows signal to freely create copies of shared_ptr instead of copying boost::python::object without the GIL.
This GIL safety slot can be encapsulated in a helper class.
/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
/// * Caller must own GIL when constructing py_slot, as
/// the python::object will be copy-constructed (increment
/// reference to the object)
/// * The newly constructed python::object will be managed
/// by a shared_ptr. Thus, it may be copied without owning
/// the GIL. However, a custom deleter will acquire the
/// GIL during deletion.
/// * When py_slot is invoked (operator()), it will acquire
/// the GIL then delegate to the managed python::object.
struct py_slot
{
public:
/// @brief Constructor that assumes the caller has the GIL locked.
py_slot(const boost::python::object& object)
: object_(
new boost::python::object(object), // GIL locked, so copy.
[](boost::python::object* object) // Delete needs GIL.
{
gil_lock lock;
delete object;
}
)
{}
// Use default copy-constructor and assignment-operator.
py_slot(const py_slot&) = default;
py_slot& operator=(const py_slot&) = default;
template <typename ...Args>
void operator()(Args... args)
{
// Lock the GIL as the python object is going to be invoked.
gil_lock lock;
(*object_)(args...);
}
private:
boost::shared_ptr<boost::python::object> object_;
};
A helper function will be exposed to Python to help adapt the types.
/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
MyClass& self,
boost::python::object object)
{
py_slot slot(object); // Adapt object to a py_slot for GIL management.
// Using a lambda here allows for the args to be expanded automatically.
// If bind was used, the placeholders would need to be explicitly added.
self.connect_slot([slot](Args... args) mutable { slot(args...); });
}
And the updated binding expose the helper function:
python::class_<MyClass, boost::shared_ptr<MyClass>,
boost::noncopyable>("MyClass")
.def("connect_slot", &MyClass_connect_slot<int>)
.def("event", &MyClass::event)
// ...
;
The thread itself.
The thread's functionality is fairly basic: it sleeps then invokes the signal. However, it is important to understand the context of the GIL.
/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
boost::shared_ptr<MyClass> self,
unsigned int seconds,
Args... args)
{
// Sleep without the GIl.
std::this_thread::sleep_for(std::chrono::seconds(seconds));
// We do not want to hold the GIL while invoking or copying
// C++-specific slots connected to the signal. Thus, it is the
// responsibility of python slots to manage the GIL via the
// py_slot wrapper class.
self->event(args...);
}
/// @brief Function that will be exposed to python that will create
/// a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
boost::shared_ptr<MyClass> self,
unsigned int seconds,
Args... args)
{
// The caller may or may not have the GIL. Regardless, spawn off a
// thread that will sleep and then invoke an event on MyClass. The
// thread will not be joined so detach from it. Additionally, as
// shared_ptr is thread safe, copies of it can be made without the
// GIL.
std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
.detach();
}
Note that MyClass_event_in_thread could be expressed as a lambda, but unpacking a template pack within a lambda does not work on some compilers.
And the MyClass bindings are updated.
python::class_<MyClass, boost::shared_ptr<MyClass>,
boost::noncopyable>("MyClass")
.def("connect_slot", &MyClass_connect_slot<int>)
.def("event", &MyClass::event)
.def("event_in", &MyClass_event_in<int>)
;
The final solution looks like this:
#include <thread> // std::thread, std::chrono
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/signals2/signal.hpp>
/// @brief Mockup class.
class MyClass
{
public:
/// @brief Connect a slot to the signal.
template <typename Slot>
void connect_slot(const Slot& slot)
{
signal_.connect(slot);
}
/// @brief Send an event to the signal.
void event(int value)
{
signal_(value);
}
private:
boost::signals2::signal<void(int)> signal_;
};
/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
gil_lock() { state_ = PyGILState_Ensure(); }
~gil_lock() { PyGILState_Release(state_); }
private:
PyGILState_STATE state_;
};
/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
/// * Caller must own GIL when constructing py_slot, as
/// the python::object will be copy-constructed (increment
/// reference to the object)
/// * The newly constructed python::object will be managed
/// by a shared_ptr. Thus, it may be copied without owning
/// the GIL. However, a custom deleter will acquire the
/// GIL during deletion.
/// * When py_slot is invoked (operator()), it will acquire
/// the GIL then delegate to the managed python::object.
struct py_slot
{
public:
/// @brief Constructor that assumes the caller has the GIL locked.
py_slot(const boost::python::object& object)
: object_(
new boost::python::object(object), // GIL locked, so copy.
[](boost::python::object* object) // Delete needs GIL.
{
gil_lock lock;
delete object;
}
)
{}
// Use default copy-constructor and assignment-operator.
py_slot(const py_slot&) = default;
py_slot& operator=(const py_slot&) = default;
template <typename ...Args>
void operator()(Args... args)
{
// Lock the GIL as the python object is going to be invoked.
gil_lock lock;
(*object_)(args...);
}
private:
boost::shared_ptr<boost::python::object> object_;
};
/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
MyClass& self,
boost::python::object object)
{
py_slot slot(object); // Adapt object to a py_slot for GIL management.
// Using a lambda here allows for the args to be expanded automatically.
// If bind was used, the placeholders would need to be explicitly added.
self.connect_slot([slot](Args... args) mutable { slot(args...); });
}
/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
boost::shared_ptr<MyClass> self,
unsigned int seconds,
Args... args)
{
// Sleep without the GIL.
std::this_thread::sleep_for(std::chrono::seconds(seconds));
// We do not want to hold the GIL while invoking or copying
// C++-specific slots connected to the signal. Thus, it is the
// responsibility of python slots to manage the GIL via the
// py_slot wrapper class.
self->event(args...);
}
/// @brief Function that will be exposed to python that will create
/// a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
boost::shared_ptr<MyClass> self,
unsigned int seconds,
Args... args)
{
// The caller may or may not have the GIL. Regardless, spawn off a
// thread that will sleep and then invoke an event on MyClass. The
// thread will not be joined so detach from it. Additionally, as
// shared_ptr is thread safe, copies of it can be made without the
// GIL.
// Note: MyClass_event_in_thread could be expressed as a lambda,
// but unpacking a template pack within a lambda does not work
// on some compilers.
std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
.detach();
}
BOOST_PYTHON_MODULE(example)
{
PyEval_InitThreads(); // Initialize GIL to support non-python threads.
namespace python = boost::python;
python::class_<MyClass, boost::shared_ptr<MyClass>,
boost::noncopyable>("MyClass")
.def("connect_slot", &MyClass_connect_slot<int>)
.def("event", &MyClass::event)
.def("event_in", &MyClass_event_in<int>)
;
}
And a testing script:
from time import sleep
import example
def spam1(x):
print "spam1: ", x
def spam2(x):
print "spam2: ", x
c = example.MyClass()
c.connect_slot(spam1)
c.connect_slot(spam2)
c.event(123)
print "Sleeping"
c.event_in(3, 321)
sleep(5)
print "Done sleeping"
Results in the following:
spam1: 123
spam2: 123
Sleeping
spam1: 321
spam2: 321
Done sleeping
