Is queuing a task always a guarantee for GAE?

Question

Is the following simple template sufficient to ensure that the task sequence never stops even after application updates or Google’s tough “intermittent” crashes.

def do_work(): ... .... deferred.defer(do_work, _countdown=..in 7 days..) 

Can I appoint such a self-adjusting employee and not look back?

Answer 1

Two answers:

Yes, the tasks will ultimately be completed and will also repeat the execution in case of errors when completing the task. Repeat parameters are set when defining a task.

No, the task queue is not a scheduler, so you cannot schedule a task to run at a specific time. Tasks placed in the task queue are immediately submitted in FIFO mode.

As @Jesse noted, for job scheduling, you should look in the GAE cron.

Answer 2

If the task is queued successfully, it will eventually be completed. (And the App Engine will continue to try as long as necessary.)

The sample you are showing may be better implemented with cron jobs , although they run the task regularly. The usual template that I use is for the daily cron task to run the task in the task queue with a small number of attempts (so if there is a temporary failure, it will immediately try again).

If you want to use the method above rather than cron, there is one more thing to worry about: since your method may be re-examined due to a crash or other system problems (for example, the instance in which it works), you should do make sure you do not have two tasks. Imagine if it started, the next task was registered, and then the node went down; App Engine will retry starting the second task. To prevent this from happening, you can use the data store (in the transaction) to check and see if the next task has already been queued. Sort of:

 def do_work(counter): ... @db.transactional def start_next(): # fetch myModel from the data store here if myModel.counter == counter: return # already started next job myModel.counter = counter myModel.put() deferred.defer(do_work, counter + 1, _transactional=True, _countdown=...) start_next() 

Note the "transactional" argument in the deferral call; this ensures that the MyModel instance is updated if and only if the next task is in the queue.

You might also want to view the email forwarding to the administrator after a certain number of failed attempts. (You can find this in the headers of the HTTP request, but you cannot use the deferred library, if you want to do this, you need to use the task queue API directly.)