How to limit the maximum number of concurrent requests?

Question

Node.js

I have an array, for example, of 2000 urls to which I need to send requests, collect responses into a new array, and after going through all the links, return the result (using Promise.all). In this case, it is necessary that no more than 20 requests be executed simultaneously. That is, the first 20 requests are sent, as soon as one of them receives a response, the next one is sent.

Answer 1

It works!!!!))
I had to write my own Promise.race, which returns the index of the resolved promise in the array.
The limit of concurrent requests is passed to the constructor, which will not exceed the function. The function makes N requests at once, where N is the limit, as soon as one promise is resolved, another request is made.

const promiseRace = promises => (
  new Promise((fulfil, reject) => {
    promises.forEach((promise, index) => {
      promise.then(data => fulfil({ data, index }), reject);
    });
  })
);

class Fetcher {
  constructor(maxConnections = 20) {
    this.maxConnections = maxConnections;
  }

  async request(urls, options = {}) {
    const responsePromises = [];

    for (const url of urls) {
      if (responsePromises.length >= this.maxConnections) {
        const { data: response, index } = await promiseRace(responsePromises);
        responsePromises.splice(index, 1);
      }
      responsePromises.push(fetch(url, options));
    }

    return Promise.all(responsePromises);
  }
}

// test
(async () => {
  try {
    const fetcher = new Fetcher(2); // лимит 2
    const urls = [];

    for (let i = 0; i < 200; i++) {
      urls.push(`/api/user/${i}`);
    }

    const responses = await fetcher.request(urls);
    console.log('end');
  } catch (e) {
    console.error(e);
  }
})();

Answer 2

If the script is run in one instance, then just a counter in memory

Answer 3

Issue a new request for each next connection on disconnect or completion of the download (and errors) of the previous one, and when starting the entire script, set the launch of 20 downloads. No counter is needed then.

Answer 4

You can try something like this

var originalArr = [...];
var firstArr = originalArr.slice(0, 19);
var restArr = originalArr.slice(19);
firstArr.forEach(fetchNext);

function fetchNext(url) {
  if (!url) {
    return;
  };
  
  fetch(url).then().catch().finaly(() => {
    fetchNext(restArr.shift());
  });
};