In a typical web application, there are some things that I would prefer to run as slow tasks / tasks . They have some or all of the following properties:
- It takes a lot of time (from several seconds to several minutes to several hours).
- Occupy some resource strongly (CPU, network, disk, external API limits, etc.).
- The result is not required immediately. Can complete an HTTP response without it. OK (and perhaps even preferable) to delay to the end.
- It may be (and perhaps preferable) to work on (a) a different machine (s) than the web server (s). Machines are potentially dedicated jobs / tasks.
- It must be executed in response to other events (events) or run periodically.
What would be the preferred way to configure, queue, schedule, and run pending tasks / tasks in Scala + Play Framework 2.x?
More details ...
The sample that I used in the past, and which I would like to reproduce, if applicable, is:
- In a web request handler or in a cron-like call, the job (s) of the queue (s)
- In runners (s) of tasks, re-delete and run one task at a time
- It is possible to process the results of a recording job
This is apparently a relatively simple but still relatively flexible template.
Examples I've come across in the past include:
- Update received data in DB
- Google Analytics / Tracking API requests for web request
- Delete expired sessions or other obsolete / obsolete database entries.
- Periodic Periodic ETL
In other languages / frameworks, I usually used the task / task structure. Examples include:
- Resque in Ruby + Rails Application
- Celery in Python + Django Application
I found the following existing materials, but, unfortunately, I do not think that they correspond to my use case directly.
- Activate the 1.x Asynchronous Jobs API (+ various SO questions that link to it). It seems that they were deleted on line 2.x. There is no reference to what replaced it.
- Play 2.x Acca integration . Seems very versatile. I would suggest that you can use Akka for the above, but I would prefer not to write a job / task structure if it already exists. In addition, there is no information on how to separate the job machine (s) from your web server (s).
- This answer is SO . This seems to be potentially promising for the case of short and medium duration IO, for example. analytical calls, but not necessary for the case of "CPU bound" (perhaps it should not bind the processor on the web server, prefers to transfer it to another node), the case is "a lot of network" or "hours", (maybe you should not leave this in the background on the web server, even if it does not have too many resources).
- This SO question and related questions . As above, it seems to me that this applies only to those cases when it would be advisable to run on the same web server.
Some additional clarifications on precedents (as requested by commentators). There are two main use cases that I experienced with something like resque or celery that I am trying to reproduce here:
- Some event on the site (Most often, an incoming web request forces a task to queue.)
- The task should be performed periodically. (Most often, this is implemented as: periodically, the task of the queue, which should be performed as described above.)
In the case of resque or celery, the tasks set in both use cases enter the queues the same way and handle the runner / worker method in the same way. Forbidding other Scala or game-related considerations, this would be my initial suggestion on how to approach this.
Some further clarification as to why I don't think Akka's planner is suitable for my use case out of the box (at the request of commentators):
Despite the fact that without a doubt you can build a suitable solution using some combination of the Akka scheduler (for periodic tasks), akka-remote and akka-cluster (for communication between the calling challenge of the task and the runner), this approach requires a certain glue code, which itself in itself is almost a delay in work. If it exists, I would rather use an existing turnkey solution rather than reinvent the wheel.