Limit the number of concurrent HTTP requests in node js

Question

Hi guys I'm new to node js.

I have a script that I run several HTTP requests inside a loop. Let's say I have to make 1000 http requests. The thing is, I can only make one HTTP request per IP address, and I only have 10 IP addresses

So, after 10 concurrent requests, I have to wait for an answer to make another one.

How can I wait without blocking the script for one response from the http request to start another?

My problem is that if I execute some time waiting for a free IP address, my whole script is blocked and I don't get any response.

Thanks.

Answer 1

Use the async module.

You can use async#eachLimit to limit concurrent requests to 10.

 var urls = [ // a list of 100 urls ]; function makeRequest(url, callback) { /* make a http request */ callback(); // when done, callback } async.eachLimit(urls, 10, makeRequest, function(err) { if(err) throw err; }); 

This code will scroll through the list of URLs and call makeRequest for each of them. He will dwell on 10 simultaneous requests and will not continue the 11th request until one of the first 10 is completed.