How many blobs may be submitted to GAE blobstore in one call?

Question

I am trying to upload 1744 small files to the blobstore (total size of all the files is 4 MB) and get HTTP/1.1 503 Service Unavailable error.

This is 100% reproducible.

Is it a bug, I do I violate any constraints? I don't see any constraints in the documentation about number of blobs submitted in one call.

Answer 1

So, the answer. Currently only < 500 files may be submitted in one request. This is going to be fixed in the scope of the ticket http://code.google.com/p/googleappengine/issues/detail?id=8032 so that unlimited number of files may be submitted. But it may take a GAE release or 2 before the fix is deployed.

Answer 2

The answer that claims that create_upload_url can only accept one file per upload above is wrong. You can upload multiple files in a single upload and this is the way you should be approaching your problem.

That being said, there was a reliability problem when doing a batch upload that was worked on and fixed around a year or so ago. If possible I would suggest keeping the batch sizes a little smaller (say 100 or so files in a batch). Each file in the batch results in a write to the datastore to record the blob key so 1744 files == 1744 writes and if one of them fails then your entire upload will fail.

If you give me the app_id I can take a look at what might be going wrong with your uploads.