Deferred tasks create new instances that cannot access some python modules

Question

I am using the latest version of GAE with auto-scaling, endpoint APIs and deferred.defer () tasks.

The problem is that from the moment the API is added, some instances will automatically deploy, which always throws constant task failures:

Permanent failure attempting to execute task Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post self.run_from_request() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request run(self.request.body) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run raise PermanentTaskFailure(e) PermanentTaskFailure: No module named app.Report 

Constant task failures are unique to one instance, though, in which all pending tasks on that instance fail. These deferred tasks throw the same error, although the tasks do not use the Api.py module. In other cases, the same deferred tasks will perform just fine if they are not redirected to the damaged instance.

The app.yaml handlers look like this:

 handlers: # Api Handler - url: /_ah/api/.* script: main.api - url: /_ah/spi/.* script: main.api # All other traffic - url: .* script: main.app builtins: - deferred: on 

The main.py file looks like this:

 import Api, endpoints, webapp2 api = endpoints.api_server([Api.AppApi]) app = webapp2.WSGIApplication( [(misc routes)] ,debug=True) 

Api.py looks like this:

 import endpoints from protorpc import messages from protorpc import message_types from protorpc import remote from google.appengine.ext import deferred from app.Report import ETLScheduler @endpoints.api(...) class AppApi(remote.Service): @endpoints.method(...) def reportExtract(self, request): deferred.defer( ETLScheduler, params ) 

I am not making any changes to the path, so I am curious why the new instance has problems finding python modules for the API, although pending tasks are in another module using other functions. Why is it throwing these errors only for this instance?

Edit:

So, after looking at some other SO issues, I tried changing the path in appengine_config.py . I moved all my folders to the lib directory and added this to the configuration file:

 import os,sys sys.path.append(os.path.join(os.path.dirname(__file__), 'lib')) 

Now the error I get in case of failure:

 Permanent failure attempting to execute task Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post self.run_from_request() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request run(self.request.body) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run raise PermanentTaskFailure(e) PermanentTaskFailure: cannot import name ETLScheduler 

Thus, it looks like it is in a module, but the same as before, none of the deferred instance tasks can import this method.

Answer 1

So, I figured out a way to make it work, but I'm not sure why it works.

When importing an entire module, and not a method from a module, new instances that are unwound for deferred tasks no longer cause a PermanentTaskFailure: cannot import name ETLScheduler .

I tried to import the whole module instead of the method, so Api.py looks like this:

 import endpoints from protorpc import messages from protorpc import message_types from protorpc import remote from google.appengine.ext import deferred # Import the module instead of the method #from app.Report import ETLScheduler import app.Report @endpoints.api(...) class AppApi(remote.Service): @endpoints.method(...) def reportExtract(self, request): deferred.defer( app.Report.ETLScheduler, params ) 

Now I no longer get instances that throw PermanentTaskFailure: cannot import name ETLScheduler . Perhaps this is a circular dependency on importing Api.py into main.py (I'm not sure), but at least it works now.

Answer 2

You miss _target kwarg in deferring if you are trying to run something in a specific module.

 deferred.defer( app.Report.ETLScheduler, params, _target="modulename" )