Sending message to RabbitMQ (pika) from scheduler callback doesn't work

Question

I need different messages to be sent to the queue - each by its personal schedule. So I have message list and related interval to resend each one. I use rabbitMQ/pika and apscheduler. According to numerous examples, I created the simplest BlockingConnection/channel/queue. When immediately after that I try to push messages - everything works fine, I can see in rabbitmq web-interface that all the messages become in the queue. Here is the piece of code that works:

    self.cr = Queue('DIRECT_C_QUEUE', True, ex_type='direct')
    for i in range(1,10000):
        self.cr.channel.basic_publish(exchange='', routing_key='DIRECT_C_QUEUE', body='hello_world')

But if I try to push messages (in exactly same way) via apscheduler callback function - only few (about 1-10) messages appear in the queue (but callbacks are fired all the time and there are no any exception when publishing message!).

Finally I begin to receive such warnings:

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pika/connection.py:642: UserWarning: Pika: Write buffer exceeded warning threshold at 1125 bytes and an estimated 43 frames behind warn(message % (self.outbound_buffer.size, est_frames_behind))

and still no new messages in the queue.

I am new in python, any help is much appreciate.

Answer 1

I found source of the problem: apscheduler runs basic_publish calls in separate thread, and pika doesn't recommend share connections between threads - http://pika.github.com/faq.html

So I had choice either create new connection each time, or put new messages in some queue and publish them from the main thread (where connection was created).

Answer 2

I fixed that problem by increasing ulimit

edit /etc/default/rabbitmq-server and set

ulimit -n 4096

then restart rabbitmq

sudo /etc/init.d/rabbitmq-server restart