Is it possible to do a nonblocking bulk insert into Mongo from Python?

Question

I have to read some data coming across the network, and then insert it into a MongoDB. All of this is done using a Python script. Is there any way to do a nonblocking insert into Mongo from Python so I can continue to process the data coming over the network while the Mongo server handles the insert?

Answer 1

Yes. There are three possibilities:

1. Do an unacknowledged insert: collection.insert(documents, w=0). The call to insert() returns as soon as the data has been written to the socket's output buffer, and your code does not block awaiting acknowledgment from MongoDB. You'll get no error reporting to your application. More info about write concerns here and here.
2. Add batches of documents to a Queue, and in a separate thread pop from the queue and insert into MongoDB: collection.insert(q.get()).
3. Port your application to Tornado and Motor as James Mills suggested; this seems to require the most learning of the three and is probably overkill.

Answer 2

Give Motor a try. It is an Asynchronous Mongo Drivers for the Tornado Web Framework.

Otherwise there are other options:

• https://pypi.python.org/pypi/asyncmongo
• https://github.com/fiorix/mongo-async-python-driver