Slow facebook API via Python on Google App Engine (GAE)

Question

I am fetching data from my news stream to filter it. This takes Facebook sometimes more than 5 seconds. I hit the url_fetch() timeout of Google App Engine.

Now is there any way to work around this timeout or to improve the speed with which Facebook replies to my request? This is the part where I get my exceptions:

params[u'access_token'] = self.access_token

result = json.loads(
  urlfetch.fetch(
    url=u'https://graph.facebook.com/me/home?limit=1000,
    payload=urllib.urlencode(params),
    method=urlfetch.POST,
    headers={u'Content-Type': u'application/x-www-form-urlencoded'}
  ).content)

Answer 1

There's nothing you can do to speed it up - how fast it is is up to facebook. You can pass the deadline argument to URLFetch to set the maximum deadline for requests (in seconds). If you're doing a lot of calls, you probably want to look into using the asynchronous API to do calls in parallel.

Answer 2

I had a similar problem with a different project. You can use the mechanize library very adequately in GAE and it allows you to specify timeouts. Just copy the folder into your GAE project and you're good to go.

Use it sparingly though as long waits really drives up costs.