Background processes in Node.js

Question

What is a good approach to handling background processes in a NodeJS application?

Scenario : after the user sends something to the application, I want to crunch the data, request additional data from external resources, etc. This all takes a lot of time, so I want it to be from the req / res loop. It would be ideal to simply have a job queue where you can quickly reset a task, and a daemon or runner will always accept the oldest and process it.

In RoR, I would do this with something like a Delayed Job. What is the Node equivalent of this API?

Answer 1

If you want something lightweight that works in the same process as the server, I highly recommend Bull . It has a simple API that allows fine control over queues.

If you are looking for something that works as a standalone workflow, maybe check out Kue . It can work as a RESTful API server, and even there are several front-end applications for it.

If you are familiar with Ruby Resque, there is a node implementation called Node-resque

Bull, Kue and Node -resque are supported by Redis , which is ubiquitous among the Node.js. work queues. All 3 could do what RoR DelayedJob does, it depends on the specific features you want and your API settings.

Answer 2

Background jobs are not directly related to the web service, so they should not be in the same process. As you scale up, the use of background job memory will affect web service performance. But you can put them in the same code repository if you want, which makes sense.

One of the good messaging options between the two processes would be redis if you drop the message every time now. If you want a “message left”, you will need a heavier broker, such as Rabbit . The web service process can be published, and your background work process can subscribe.

Two processes do not require co-location; they can be on separate virtual machines, Docker containers, regardless of what you use. This allows you to significantly reduce the scale.

Answer 3

If you are using MongoDB, I recommend Agenda . Thus, individual instances of Redis do not start, and all functions, such as scheduling, ordering and the web interface, are present. The Agenda user interface is optional and can be run separately, of course.

I would also recommend installing a loosely coupled abstraction between your application logic and the priority / scheduling system, so that you can change the entire background processing system if necessary. In other words, keep so much application logic and processing from your task definitions on the Agenda so that they are easy.

Answer 4

I would suggest using Redis for job scheduling. It has many different data structures, you can always choose the one that is best suited for your use.

You mentioned RoR and DJ, so I assume you are familiar with sidekiq. You can use node-sidekiq to schedule tasks if you want, but its suboptimal imo, since the main goal is to combine nodejs with RoR.

To demonstrate the employee, I recommend using PM2 . It is widely used and actively supported. It solves many problems (for example, deployment, monitoring, clustering), so make sure that it is not unnecessary for you.

Answer 5

I suggest using the correct Node.js framework to build your application.

I think the most powerful and easiest to use Sails.js .

This is the structure of MVC, so if you are used to developing in ROR, it will be very easy for you!

If you use it, it already is a powerful (in javascript terms) task manager.

new sails.cronJobs('0 01 01 * * 0', function () { sails.log.warn("START ListJob"); }, null, true, "Europe/Dublin"); 

If you need more information, feel free to contact me!