How can I fetch() more than 1MB on Google App Engine?

Question

Google App Engine limits urlfetch.fetch() responses to 1MB. Is there any workaround of this (switching to paid version maybe)?

I'm using Python and if it's possible to provide an example that would be great.

Answer 1

With the brand new SDK 1.4.0 you can download 32MByte; keep in mind that you still have the 10 seconds Deadline limit though ;-) . deadline can be up to a maximum of 60 seconds for request handlers and 10 minutes for tasks queue and cron job handlers.

URLFetch allowed response size has been increased, up to 32 MB. Request size is still limited to 1 MB.

Answer 2

No, you cannot fetch more than 1MB per URL fetch (even if you enable billing). However, you might be able to fetch portions of the target URL using the Range header and then combine these pieces. This might even be faster since you could fetch each 1MB chunk simultaneously (using asynchronous fetches).