Google App Engine parsing website locally and online

Question

I'm using Google App Engine with Python and I want to parse this website and get the displayed data, that changes over time. The problem is: the application works perfect when I'm running it locally but it doesn't (displays the same old information) when it's running online. I'm using a library called urllib and I don't get any error, just the same old data. I can't figure out what's happening so I would appreciate some ideas.

Thanks

Answer 1

This is because App Engine caches http requests.

I am not familiar with urllib, but with urlfetch from google.appengine.api to ensure you are getting the content fresh from the website you need to set the Cache-Control header, in the example below it will only return a cached page if it is less than 30 seconds old, otherwise it will go to the website to get the page fresh.

content = urlfetch.fetch(url, headers = {'Cache-Control' : 'max-age=30'})