Google App Engine - Task Dependency

Question

In my application, I have a long task, so I broke it into n smaller tasks. After completing these n tasks, it is necessary to complete one more task, and this depends on the results of these n tasks. How to achieve this dependency using Task API? that is, perform one task after the other n tasks.

Answer 1

I think there are 2 methods that can solve this problem. Suppose that task TD depends on n other tasks TA and there is a queue Q.

• Push n TA tasks into the Q queue. When each TA task finishes, it checks to see if the last one in the Q queue is itself. If TA is the last task in the Q queue, it pushes TD to the Q queue.

• Press n TA and TD tasks for queue Q. When the TD starts, it checks to see if all TA work has completed. If any TA is not finished, TD cancels its execution, returning any HTTP status code outside the range of 200-299.

The key to these methods is to get the number of tasks in the Q queue. Although I have not tried it, I know that there is a Python API that provides an experimental method to get the TaskQueue resource of a specific queue. The stats.totalTasks property is the total number of queues in the queue.

See http://code.google.com/intl/en/appengine/docs/python/taskqueue/rest.html

Answer 2

Take a look at the GAE Pipeline API , it is used to create a complex workflow as described by you.

Answer 3

Another approach might be to add all tasks to the queue. Before you finish, enter the N primary task log information in the data store, in some way, so that you can query the data store to make sure that they are all running.
When a dependent task is launched, it executes this data warehouse request to check if its conditions are met (it is checked that all initial tasks are registered, that they are completed). If not, you need to run it later.
To do this, the dependent task could add a copy to the queue scheduled to run after a certain period of time. Alternatively (as in the answer above), the dependent task may end with an error status code, in which case it will be automatically repeated at some later point if the retry_limit for the queue or task is not exceeded.