How to execute / abort long tasks in Node JS?

Question

NodeJS server with Mongo DB - one function will generate a JSON file report from the database, which may take some time (60 seconds - process hundreds of thousands of records).

We want to run this as a background job. We need to be able to start the process of building the report, control it and interrupt it if the user decides to change the parameters and restore it.

What is the easiest approach with node? Actually, you don’t want to get into the areas of processing work servers of individual work groups, message queues, etc. - we need to save this in the same field and rather simple implementation.

1) Run the assembly as an asynchronous method and return to the user using the socket.io release report?

2) Release the child process to build the script?

3) Use something like https://www.npmjs.com/package/webworker-threads ?

With several approaches that I have looked at, I am stuck in the same two areas;

1) How to track progress? 2) How to interrupt the existing build process if the user retransmits the data?

Any pointers would be greatly appreciated ...

Answer 1

It’s best to separate this task from the main application. However, it would be easy to run it in the background. To run it in the background and save it without a message queue, etc., child_process will be the child_process .

• You can run the spawn task at the endpoint (or url) that the user invokes.
• Then configure a socket to return live monitoring of the child process
• Add another endpoint to stop the job, with a unique identifier returned by 1. (or not, depending on your concurrency needs)

Some coding ideas:

 var spawn = require('child_process').spawn var job = null //keeping the job in memory to kill it app.get('/save', function(req, res) { if(job && job.pid) return res.status(500).send('Job is already running').end() job = spawn('node', ['/path/to/save/job.js'], { detached: false, //if not detached and your main process dies, the child will be killed too stdio: [process.stdin, process.stdout, process.stderr] //those can be file streams for logs or wathever }) job.on('close', function(code) { job = null //send socket informations about the job ending }) return res.status(201) //created }) app.get('/stop', function(req, res) { if(!job || !job.pid) return res.status(404).end() job.kill('SIGTERM') //or process.kill(job.pid, 'SIGTERM') job = null return res.status(200).end() }) app.get('/isAlive', function(req, res) { try { job.kill(0) return res.status(200).end() } catch(e) { return res.status(500).send(e).end() } }) 

To execute the child process, you can use pidusage , we use it in PM2 . Add a route to complete the task and call it every second. Remember to free up memory when completing a job.

---

You might want to check out this library , which will help you manage multiprocessing through microservices.