My application has to perform many data warehouse operations for each request. I would like to run them in parallel to get the best response time.
For data warehouse updates, I do batch interference, so they all happen asynchronously, which saves many milliseconds. App Engine allows you to update up to 500 objects in parallel .
But I did not find a built-in function that allows you to select data of different types in parallel.
Since App Engine allows urlfetch calls to run asynchronously , I created a recipient URL for each type that returns the query results as JSON-formatted text. Now my application can make asynchronous urlfetch calls for these URLs, which can parallelize data store samples.
This method works well with a small number of concurrent requests, but the App Engine causes errors when trying to run more than 5 or 10 of these URL calls at the same time.
I am testing only now, so each urlfetch is an identical request; since they work fine in small volumes, but start working with more than a few simultaneous requests, I think this should have something to do with async urlfetch calls.
My questions:
- Is there a limit on the number of urlfetch.create_rpc () calls that can be executed asynchronously?
- The synchronous urlfecth.fetch () function has a deadline parameter that allows the function to wait up to 10 seconds to respond before a failure. Is there any way to tell urlfetch.create_rpc () how long to wait for an answer?
- What the following errors mean:
Is there a more efficient server-side technology for running different types of data warehouse samples in parallel?
"/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", 501, get_result return self.__ get_result_hook (self) "/base/python_lib/versions/1/google/appengine/api/urlfetch.py", 331, _get_fetch_result raise DownloadError (str (err))
InterruptedError: (' Wait() :', DownloadError ('ApplicationError: 5',))