Is the Google appengine cache an external request?

Question

I have a very simple appengine application that requests a web page every five minutes and analyzes a specific piece of data.

Everything works fine, except that the response I received from an external request (using urllib2) does not reflect the latest changes on the page. Sometimes it takes a few minutes to get the last, sometimes more than an hour.

Is there a transparent caching layer that appengine adds? Or is something else missing here? I looked at the caching headers of the requested page and did not send Expires, LastModified or ETags headers.

Update: Sometimes it gets a new version of a page for multiple requests, and then accidentally gets an old obsolete version.

Answer 1

This seems to be a problem that the App Engine team is aware of. The proposed solution is to set the Cache-Control header with max-age in seconds:

result = urlfetch.fetch(url, headers = {'Cache-Control' : 'max-age=240'})

hope it works for you.