Celery: speed limit for tasks with the same parameters

Question

I am looking for a way to limit when calling a function, but only when the input parameters differ, that is:

@app.task(rate_limit="60/s") def api_call(user): do_the_api_call() for i in range(0,100): api_call("antoine") api_call("oscar") 

Therefore, I would like api_call("antoine") called 60 times per second and api_call("oscar") 60 times per second .

Any help on how I can do this?

- EDIT 04/27/2015 I tried to call the subtask with rate_limit inside the task, but it also does not work: Rate_limit is always applied to all created subtasks or tasks (which is logical).

 @app.task(rate_limit="60/s") def sub_api_call(user): do_the_api_call() @app.task def api_call(user): sub_api_call(user) for i in range(0,100): api_call("antoine") api_call("oscar") 

Best!

Answer 1

I do not think that this can be achieved using the Celery built-in task limiter.

Assuming you are using some kind of cache for your API, the best solution would be to create a hash of the task name and arguments and use this key for a cache-based throttle.

If you use Redis, you can either set a lock with a 60 second timeout, or use an incremental counter to count calls per minute.

This post may give you some pointers to distributed Celery task control using Redis:

https://callhub.io/blog/2014/02/03/distributed-rate-limiting-with-redis-and-celery/