GAE (Python) Urlfetch 60 Second Timeout Issue

Question

I ran into this 60s timeout issue when my GAE server communicates to an EC2 REST server. On the GAE side, my task is to uploading a csv file, parsing its information and sending each row as a request to the REST server. I am using multiple threading and task queue to reduce the request time but still got timed-out when the file is large. Below is an example of my code and I appreciate any suggestions.

from threading import Thread
import Queue

thread_count = 10 #the number of theading
job_q = Queue.Queue() #a job queue

def html_table(row_inp_all):
    while True:
        row_inp_temp_all = row_inp_all.get()
        all_dic = {"row_inp_temp_all": row_inp_temp_all}
        data = json.dumps(all_dic)
        url=url_part1 + '/przm/' + jid 
        response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers, deadline=60)   
        output_val = json.loads(response.content)['result']


def loop_html(thefile):
    reader = csv.reader(thefile.file.read().splitlines())
    header = reader.next()
    for row in reader:
        job_q.put(row)
    all_threads = [Thread(target=html_table, args=(job_q, )) for j in range(thread_count)]
    for x in all_threads:
        x.start()
    for x in all_threads:
        job_q.put(None)
    for x in all_threads:
        x.join()

Answer 1

There is a hard limit for any request of 60 seconds and if you want more time than that you should use Task Queues API which can run in the background for up to 10 minutes which should be more than enough in your case.

Before diving deep into the Task Queues you could try the background work with the deferred library which is actually using Task Queue API but makes it much easier to work with.