gevent and twisted producer performance problems

Question

I'm writing a minimal server to spit out a never ending stream of sensor data using twisted.

The sensor data capture library uses gevent and an internal gevent.Queue to store the data that I'm serving up.

For some reason, data is sent in a painfully slow and seemingly buffered fashion. I've checked that data is generated in an even and high pace from the sensor, so that's not the problem. Replacing the dequeuing of real data in the twisted producer and just setting it to a constant also works fine - so it has to be related to how gevent and twisted interact.

The code to setup the data producing sensor looks like this (in the producer constructor):

gevent.spawn(self._headset.setup)
gevent.sleep(1)

And the twisted resumeProducing function looks like this:

def resumeProducing(self):
    self._paused = False
    while not self._paused:
        packet = self._headset.dequeue()
        sensors = packet.sensors
        data_str = ",".join(str(sensors[x]['value']) for x in sensor_names)
        self._proto.transport.write(data_str + '\n')

Answer 1

I guess the gevent loop is not running properly, you should try the gevent reactor for Twisted:

http://wiki.inportb.com/wiki/Projects:Python:Geventreactor

It is as easy as inserting those 2 lines before you import twisted:

import geventreactor
geventreactor.install()