How to send 10000 requests via setTimeout in Node.JS with adequate memory consumption?

Question

JavaScript

There is a tabular service Airtable (a la Excel), with the ability to write rows to a table via the API. You need to roll into this service line by line 10,000 lines from the Data collection . There are API limits of 10 requests/sec. I do this placement via a view construct (setTimeout/setInterval):

Data.forEach(function(item,index){
    setTimeout(function(){
        API.sendData(item);
    });
});

The problem is, as I understand it, that forEach quickly processes the entire collection and pushes this series of functions into memory (or in some other way systematically and completely eats it out). As a result, I don’t have enough to transfer 20MB of a database in 10,000 records of 1.5Gb of available server memory and the application crashes and / or does not process the collection, this is also not very clear: whose fault is it - mine or the API gives timeout from above and my requests hang longer.
It seems to me that the above construction is simply not suitable for this kind of task, and, for sure, you might have come across a similar one in your experience. Need advice and guidance. Thank you.
UPD : API code and server response
Code

var Airtable = require('airtable');
var app = new Airtable({ apiKey: 'YOUR_API_KEY' }).app('appMzaconT6Iw9A3g');

app('Categories').create({
  "Category name": "Social Innovation",
  "Number of Orgs.": " 59",
  "CatID": "bf5c955e6f68da36e5021d3962c28325",
  "Last update": "2015-03-11T09:08:44.000Z"
}, function(err, records, newOffset) {
    if (err) { console.log(err); return; }
    records.forEach(function(record) {
        console.log(record);
    });
});

Output

{
    "id": "rec00GC0bR7Eh1bGv",
    "fields": {
        "Category name": "Social Innovation",
        "Number of Orgs.": " 59",
        "CatID": "bf5c955e6f68da36e5021d3962c28325",
        "Last update": "2015-03-11T09:08:44.000Z"
    }
}

Answer 1

I would do something like this:

var l = Data.length;
var n = 0;

function send() {
  API.sendData(Data[n]);
  n++;
  if (n < l) {
    setTimeout(send, 100);
  }
}

send();

The meaning is simple, we do not queue all 10,000 requests at once, but add one to the queue after the previous one is completed. Then you will always have only one request pending.

Answer 2

var i = 0;
var data = [];
function foo() {
  if (i == data.length) 
      return;
  // send one item to airtable
  i++;
  setTimeout(foo, 1000);
}

+ you can open an issue for their own library https://github.com/airtable/airtable.js

Answer 3

sending by 1 item? why not send 500?

Answer 4

I think it's most logical to use async

async.eachLimit(records, 8, function(record, async_cb) {
   /// your code
   async_cb();
}, function() {
 console.log('done!');
});

Will make 8 requests in parallel, after the completion of the previous one, a new request will be launched

Answer 5

I know this is an old question, but I think this is the easier way.

function sleep(time) {
    return new Promise((resolve, reject) => {
        setTimeout(resolve, time)
    })
}

and here is an example of usage

async function cycle() {
    const timeToSleep = 1000
    console.log('start')
    await sleep(timeToSleep)
    for (const value of [0,1,2,3,4,5,6]) {
        console.log(value)
        await sleep(timeToSleep)
    }
    console.log('stop')
}