Django - Share a RabbitMQ Connection between All Processes that apache creates per request?

StackOverflow https://stackoverflow.com/questions/13055718

Domanda

My stack is Django 1.3(Python 2.7) + Apache + mod_wsgi

Apache spawns a new process per request... Sometimes in those requests I have to Publish a message to RabbitMQ about some heavy task to be done later (like notifying someone's followers about a cat photo they just posted).

Publishing involves making connection to RabbitMQ, which is expensive. (see this SO question: RabbitMQ create connection is an expensive operation)

I am also sadly aware of Python's incapability to share memory across processes... sigh :(

How in this case, do I prevent the overhead of creating and closing a RabbitMQ connection per apache request?

For Clarity, Using RabbitMQ + pika here is my failed attempt at sharing the connection between processes:

def test_view(request):
    """a django view publishing to RabbitMQ"""
    connection = Connection.get_shared_connection()
    channel = connection.channel() 
    channel.queue_declare(queue='test_queue', auto_delete=False, durable=True)
    channel.basic_publish(exchange='', routing_key='test_queue', body='Hello Rabbit!', 
                                properties=pika.BasicProperties(delivery_mode=2))
    print 'SENT:', body

    return HttpResponse("published")


class Connection(object):
    """failed attempt to create a singleton"""
    _connection = None

    @staticmethod
    def get_shared_connection(): # return a pika connection
        if not Connection._connection:
            print 'NEW CONNECTION'  # prints NEW CONNECTION with every request
            Connection._connection = pika.BlockingConnection(pika.ConnectionParameters(
                                                                           host='localhost'))
        return Connection._connection

In server setting (Django + Apache + mod_wsgi) 'NEW CONNECTION' is printed with every request made to test_view

It seems like an issue anybody would run into using Django with RabbitMQ. There has to be a way...

Thanks... any help is appreciated

È stato utile?

Soluzione

Apache does NOT spawn a new process per request.

You might be getting deceived into thinking it is because in embedded mode it can be a multiprocess web server, thus different requests could be getting handled in a separate process.

I would have suggested you use mod_wsgi daemon mode and use a single multithreaded process, but unfortunately your code is not thread safe as is because you don't protect creation of your global socket connection from multiple threads at the same time. But then, that alone will not help because if multiple threads try and use that one connection at the same time, it will likely cause problems. So you really need to have a connection object per thread.

I suggest you read the following to get some background on different process/thread models of Apache/mod_wsgi.

Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow
scroll top