DBus-Cherrypy merge issue

Question

I'm using python-dbus and cherrypy to monitor USB devices and provide a REST service that will maintain status on the inserted USB devices. I have written and debugged these services independently, and they work as expected.

Now, I'm merging the services into a single application. My problem is: I cannot seem to get both services ( cherrypy and dbus ) to start together. One or the other blocks or goes out of scope, or doesn't get initialized.

I've tried encapsulating each in its own thread, and just call start on them. This has some bizarre issues.

class RESTThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        cherrypy.config.update({ 'server.socket_host': HVR_Common.DBUS_SERVER_ADDR, 'server.socket_port': HVR_Common.DBUS_SERVER_PORT, })
        cherrypy.quickstart(USBRest())

class DBUSThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        DBusGMainLoop(set_as_default=True)
        loop = gobject.MainLoop()
        DeviceAddedListener()
        print 'Starting DBus'
        loop.run()

print 'DBus Python Started'
if __name__ == '__main__':
    # Start up REST

    print 'Starting REST'
    rs = RESTThread()
    rs.start()

    db = DBUSThread()
    db.start()

    #cherrypy.config.update({ 'server.socket_host': HVR_Common.DBUS_SERVER_ADDR, 'server.socket_port': HVR_Common.DBUS_SERVER_PORT, })
    #cherrypy.quickstart(USBRest())

    while True:
        x = 1

When this code is run, the cherrypy code doesn't fully initialize. When a USB device is inserted, cherrypy continues to initialize ( as if the threads are linked somehow ), but doesn't work ( doesn't serve up data or even make connections on the port ) I've looked at cherrypys wiki pages but haven't found a way to startup cherrypy in such a way that it inits, and returns, so I can init the DBus stuff an be able to get this out the door.

My ultimate question is: Is there a way to get cherrypy to start and not block but continue working? I want to get rid of the threads in this example and init both cherrypy and dbus in the main thread.

Answer 1

I figured this out. Apparently, there are a bunch of thread contention problems in glib. If you make an app that has DBusGMainLoop in it, then you cannot create another thread in your app. The new thread blocks immediately when start() is called on it. No amount of massaging will get the new thread to run.

I found a site that had an obscure reference to dbus.mainloop.glib.threads_init(), and how this must be called before initializing a new thread. However a new problem is uncovered when this is tried. An exception is thrown that says g_thread_init() must be called before dbus.mainloop.glib.threads_init() is called. More searching uncovered another obscure reference to gobject.threads_init(). It seemed to fit, so after much experimentation, I discovered the correct sequence.

Here's the solution.

dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
gobject.threads_init()
dbus.mainloop.glib.threads_init()    
DBUSMAINLOOP = gobject.MainLoop()

print 'Creating DBus Thread'
DBUSLOOPTHREAD = threading.Thread(name='glib_mainloop', target=DBUSMAINLOOP.run)
DBUSLOOPTHREAD.start()

print 'Starting REST'
cherrypy.config.update({ 'server.socket_host': Common.DBUS_SERVER_ADDR, 'server.socket_port': Common.DBUS_SERVER_PORT, })
cherrypy.quickstart(USBRest())

Gosh what a nightmare. Now to make it better.

Answer 2

Yes; don't use cherrypy.quickstart. Instead, unpack it:

cherrypy.config.update(conf)
cherrypy.tree.mount(USBREST())
cherrypy.engine.start()

Quickstart does the above, but finishes by calling engine.block(). If your program has some main loop other than CherryPy's, omit the call to engine.block and you should be fine. However, when your foreign main loop terminates, you'll still want to call cherrypy.engine.stop():

loop = gobject.MainLoop()
try:
    loop.run()
finally:
    cherrypy.engine.stop()

There are some other gotchas, like whether CherryPy should handle Ctrl-C and other signals, and whether it should autoreload. Those behaviors are up to you, and are all fairly easy to enable/disable. See the cherrypy.quickstart() source code for some of them.