Random timeout error with Pika and gevent

Question

I've been trying to make use of RabbitMQ from within my gevent program by using the Pika library (monkey patched by gevent), gevent likes randomly throwing a timeout error.

What should I do? Is there another library I could use?

WARNING:root:Document not found, retrying primary.
Traceback (most recent call last):
  ...
  File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 32, in __init__
    BaseConnection.__init__(self, parameters, None, reconnection_strategy)
  File "/usr/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 50, in __init__
    reconnection_strategy)
  File "/usr/lib/python2.7/dist-packages/pika/connection.py", line 170, in __init__
    self._connect()
  File "/usr/lib/python2.7/dist-packages/pika/connection.py", line 228, in _connect
    self.parameters.port or  spec.PORT)
  File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
    self._handle_read()
  File "/usr/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 151, in _handle_read
    data = self.socket.recv(self._suggested_buffer_size)
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 427, in recv
    wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 169, in wait_read
    switch_result = get_hub().switch()
  File "/usr/lib/python2.7/dist-packages/gevent/hub.py", line 164, in switch
    return greenlet.switch(self)
timeout: timed out

Answer 1

Pika is not ideally suited to use with gevent because pika implements its own asynchronous connection to RabbitMQ based on non-blocking sockets. This just does not fit well with gevent's implementation of the same.

You may want to consider using py-amqplib or kombu

Answer 2

I'm also having timeout problems with using Pika in a Django/Gunicorn application. I played with raising connection_attempts or increasing the timeout but RabbitMQ always closed the connection with a handshake error. The latter seems to indicate that Pika never transmitted any data on the socket.

The cause for the timeouts could be this libevent bug - at least in my environment the script attached to the bug is able to reproduce the issue.

You could try upgrading to gevent>=1.0 (at the time of writing not released yet):

wget http://gevent.googlecode.com/files/gevent-1.0b4.tar.gz
pip install gevent-1.0b4.tar.gz