Receiving celery assignments

Question

I have a long celery job that iterates over a lot of elements and performs some actions.

The task must somehow inform which element is currently being processed, so the end user is aware of the progress of the task.

Currently my django and celery application are on the same server together, so I can use Django models to report status, but I plan to add more workers who are far from Django, so they cannot reach the database.

Now I see several solutions:

• Save intermediate results manually using some memory, such as redis or mongodb, then accessible over the network. This bothers me a bit, because if, for example, I use redis, I have to synchronize the code on the Django side, looking at the Celery status and task, writing down the status, so they use the same keys.
• Report Django status from celery using REST calls. howPUT http://django.com/api/task/123/items_processed
• Perhaps use Celery's event system and create events such as Item processedwhere django updates the counter.
• Create a separate worker that runs on the server with django, which contains a task that only increases the number items proceeded, so when the task is executed with an element, it returns increase_messages_proceeded_count.delay(task_id).

Are there any solutions or hidden problems with the ones I talked about?

Answer 1

, , , .

, django:

from django.core.cache import cache

@app.task()
def long_running_task(self, *args, **kwargs):
    key = "my_task: %s" % self.result.id
    ...
    # do whatever you need to do and set the progress
    # using cache:
    cache.set(key, progress, timeout="whatever works for you")
    ...

, , AJAX GET . - :

 def task_progress_view(request, *args, **kwargs):
     key = request.GET.get('task_key')
     progress = cache.get(key)
     return HttpResponse(content=json.dumps({'progress': progress}),
                         content_type="application/json; charset=utf-8")

, , , , - memcached, django . , , task_id , .

Answer 2

:

django - ( , )

, , .

. redis " " , currenlty.

Answer 3

- Swampdragon Celery ( , , CORS). , .

Answer 4

Take a look at flower- real-time monitor and web admin for Selery distributed task queue:

• https://github.com/mher/flower#api
• http://flower.readthedocs.org/en/latest/api.html#get--api-tasks

You need it for a presentation, right? flowerworks with websockets.

For example, to receive real-time task completion events (taken from official documents):

var ws = new WebSocket('ws://localhost:5555/api/task/events/task-succeeded/');
ws.onmessage = function (event) {
    console.log(event.data);
}

You will most likely have to work with tasks ('ws: // localhost: 5555 / api / tasks /').

Hope this helps.