Postpone multiple tasks in Google App Engine asynchronously

Question

I have a task that takes 3 parameters, a dictionary with an identifier and a title, a user key and a list of entries already in the database. defer_fetch is called in a for loop N times.

@ndb.tasklet
def defer_fetch(data, user_key, already_inserted):
    if data['_id'] not in already_inserted:
         document_key = yield Document.fetch_or_create(data)
         yield defer_fetch_document(user_key, document_key)
    else:
         document_key = alread_inserted[data['_id']]
    raise ndb.Return(document_key)

@ndb.tasklet
def defer_fetch_document(user_key, document_key):
    deferred.defer(process_document, user_key, document_key, _queue="process-documents")
    raise ndb.Return(True)

The document.fetch_or_create code runs in parallel between all defer_fetch calls, but the fetch_document request is missing, as shown in the attachment

How to make defer_fetch_document file also parallel?

Answer 1

, ndb.tasklets( deferred.defer, ). add_async.

@ndb.tasklet
def defer_fetch_document(user_key, document_key):
    queue = taskqueue.Queue("process-documents")
    task = taskqueue.Task(url="<url for worker>", 
                          params={"document_key": document_key.urlsafe()})
    yield queue.add_async(task)  #this returns a rpc which you can yield on
    raise ndb.Return(True)

, ( ), add_async .