Python Shared Libraries: RTLD_GLOBAL segfault

Question

I work with a python swig-wrapped C++ library. In it's __init__.py file, it sets the dlopen flag RTLD_GLOBAL before importing the shared object file containing the implementation code.

This causes the subsequent import of scipy.linalg to segfault, at least on my machine. I think this behavior depends on the way in which scipy was built and what it's linked against though.

# minimal example of what's going on
$ cat test.py
import sys
import ctypes
flags = sys.getdlopenflags()
sys.setdlopenflags(flags | ctypes.RTLD_GLOBAL)
import scipy.linalg

$ python test.py
[1]    16886 segmentation fault (core dumped)  python test.py

1. Why does this happen? What is going on?
2. Under what conditions might setting RTLD_GLOBAL be necessary? The code that I work with contains the comment "# The following is an evil incantation that is needed to permit the POSIX "dlopen" function to work. I do not understand it. If a better solution is known, please forward to the PyOpenMM code maintainers." When I remove the sys.setdlopenflags(flags | ctypes.RTLD_GLOBAL) line, everything seems to work fine with the library, so maybe this is specific to certain python versions or platforms?

Answer 1

You are hitting a bug in the f2py tool, which is used in building SciPy. See more details here: https://github.com/numpy/numpy/issues/2521

Unfortunately, you can only fix the problem by rebuilding SciPy, or by removing the RTLD_GLOBAL flag.

What's going on is that both NumPy and SciPy are using a symbol PyArray_API, and the RTLD_GLOBAL flag forces SciPy to (attempt to) export its own copy. This leads to a conflict and a segfault. (if anyone can explain this in more detail, i'd love to know)

RTLD_GLOBAL causes symbols from shared libraries to be made public and available for relocation. This is needed when you import several separate libraries via dlopen(), that use each other's symbols. In Python, this would be the case when a single project (PyOpenMM) consists of several binary submodules that want to share common functionality provided by one of them. The fact that "everything seems to work fine" could simply mean that you aren't using anything that needs the shared features -- or that PyOpenMM actually doesn't require this anymore.