Stream file to MongoDB with PyMongo

Question

I'm trying to upload a large file into GridFS using PyMongo and an API I wrote in Tornado. Since the file size is 4.1GB Tornado throws an Content-Length too long.

This scenario works with NodeJS and this module but I am unsure if PyMongo supports file streaming.

Answer 1

PyMongo supports streaming an input file to GridFS, but Tornado does not. See issue 231. Although the original issue report is old, rumors on the Tornado mailing list suggest this is going to be fixed within the next few minor releases. (Currently at Tornado 3.2.) When Tornado sees a content-length over 100MB it raises the exception you observe.

One idea is to front your application with Nginx and use its file-upload module to stream the user's upload into a temp file. Nginx will tell your application when the file is uploaded, and you can then stream the temp file into GridFS. Avoid blocking the Tornado process while you do this, either by spawning a subprocess to copy the file into GridFS with PyMongo, or by using Motor.