PyPy cpyext: any documentation? how to use? PyThreadState_Get error?

Question

I have read (here) that PyPy has support for CPython extension modules via cpyext.

I haven't found any cpyext documentation. Is there any?

How do I use it?

From the source code (e.g. here), I figured out that to load my leveldb.so module, I probably have to do this:

import cpyext
cpyext.load_module("leveldb.so","leveldb")

However, this crashes with this error:

Fatal Python error: PyThreadState_Get: no current thread

I noticed in the backtrace that it calls the functions from my CPython, not from PyPy:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib          0x00007fff8b3e4d46 __kill + 10
1   libsystem_c.dylib               0x00007fff927a9df0 abort + 177
2   org.python.python               0x0000000104692eaa Py_FatalError + 49
3   org.python.python               0x0000000104691370 PyThreadState_Get + 28
4   org.python.python               0x000000010468cf16 Py_InitModule4_64 + 58
5   leveldb.so                      0x00000001027e0881 initleveldb + 49 (leveldb_ext.cc:59)
6   pypy                            0x0000000100f59bb3 PyLong_CheckExact + 55379
7   pypy                            0x0000000100f6e7c7 PyLong_CheckExact + 140391
....

Answer 1

I figured it out.

It is necessary to recompile the module. I must use the header files from PyPy. For linking, I must not link against libpython. I must just tell the linker to ignore about unresolved symbols.

This where my commands to build py-leveldb on MacOSX:

cc -I /usr/local/Cellar/pypy/1.9/include -g -c leveldb_ext.cc leveldb_object.cc
libtool -dynamic -o leveldb.so leveldb_ext.o leveldb_object.o -lleveldb -lsnappy -lc -lstdc++ -undefined dynamic_lookup

That worked. I just did cpyext.load_module("leveldb.so","leveldb").

I also found some more notes about CPyExt in their wiki here.

Answer 2

Plyvel, a fast and feature-rich Python interface to LevelDB, also passes almost all of its many unit tests when running under PyPy with cpyext. (The one exception is related to threading.) Installation is as simple as "pip install plyvel"; no need to fiddle with compiler flags.

See https://github.com/wbolster/plyvel and https://plyvel.readthedocs.org/ for more information.

(Disclaimer: I wrote Plyvel.)