Promise.all consumes all my RAM

Question

I have a speed limiter for the API that I use that allows 20 requests per second. All requests are based on promises, and the promise will be resolved with the API data after the response.

Problem:

I install promArray, which contains 58k promises, everyone is waiting for an answer. So slowly the memory grows until I run out of memory. In my specific situation, I do not need to pass the allowed data to my then() , and the data eats all my RAM.

Code:

  }).then(() => { // 2. Crawl for all clanprofiles from these leaderboards const promiseArray = [] for (let i = 0; i < clanTags.length; i++) { // Resolved data from getClanProfile() is eating up all my RAM const p = backgroundScheduler.getClanProfile(clanTags[i], true) promiseArray.push(p) } return Promise.all(promiseArray) }).then(() => { 

So, is there a way to wait until the promArray algorithm is resolved without the need for data resolution?

Answer 1

You will use less memory if you never have 58k promises, their associated asynchronous operations and their result data will be active immediately.

Instead, you want to start X operations right away, and then when you are done, you start the next with no more than X in flight at the same time and no more than X promises to use right away.

You can experiment with the corresponding value of X. A value of 1 is a sequential operation, but you can often improve the overall end-to-end time by using a slightly higher value of X. If all requests go to the same host, then X probably does not exceed 5-10 (since this host cannot do many things at once and ask him to do more than he can do right away, it just slows him down).

If each request relates to a different host, you can do X above. Experimentation will give you the best value for both peak memory usage and overall throughput, and to some extent depends on your specific circumstances.

Bluebird Promise.map() has a concurrency parameter that will do this for you, but there are also many ways to encode only X in flight at the same time.

Here are some other examples of in-flight quantity control coding at a time:

Make some API requests that can only handle 20 requests per minute

How to perform promises in a series?

failed to complete promises due to out of memory

Killed 100,000 requests 100 at a time

How to make it so that I can execute, for example, 10 promises at a time in javascript to prevent speed limitation on api calls?

---

If you do not need allowed data, you can let it be GCed earlier by replacing it as follows:

  const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => { return 0; // make resolved value just be a simple number // so other data is now eligible for GC }); promiseArray.push(p) 

---

And here is a simple implementation that iterates an array with no more than X queries in flight at the same time:

 // takes an array of items and a function that returns a promise // runs no more than maxConcurrent requests at once function mapConcurrent(items, maxConcurrent, fn) { let index = 0; let inFlightCntr = 0; let doneCntr = 0; let results = new Array(items.length); let stop = false; return new Promise(function(resolve, reject) { function runNext() { let i = index; ++inFlightCntr; fn(items[index], index++).then(function(val) { ++doneCntr; --inFlightCntr; results[i] = val; run(); }, function(err) { // set flag so we don't launch any more requests stop = true; reject(err); }); } function run() { // launch as many as we're allowed to while (!stop && inflightCntr < maxConcurrent && index < items.length) { runNext(); } // if all are done, then resolve parent promise with results if (doneCntr === items.length) { resolve(results); } } run(); }); }