G
G
Gorod_b2017-07-07 13:25:13
JavaScript
Gorod_b, 2017-07-07 13:25:13

How to synchronously send a request when reading a file line by line?

I've only recently been introduced to JS and asynchronous programming, so I still struggle when I encounter asynchronous code execution.
The goal is to test one service, for this I uploaded to a JSON file all the records from which the URLs for requests are formed. And you need to check each entry by sending a separate request for it. 10k records were successfully uploaded, everything is readable, but when I started reading the file line by line, I ran into a problem that requests go asynchronously to unformed URLs, and 500 errors are returned accordingly.
To send requests, I use request-promise.
For line-by-line reading byline

var user = api.webAuthorize(qaqaqa);

user.then(() => {
  var file = './assets.json';
  var stream = byline(fs.createReadStream(file, { encoding: 'utf8' }));
  stream.on('data', (line) => {
    var parsedLine = JSON.parse(line);
    api.getPlayList(parsedLine.asset).then((body) => {
      //обработка тела ответа
    });
  });
});

At the same time, I used the same construction in another place and everything worked out successfully, but the size of the chunk when reading the file there was larger than the contents of the file, I believe that because of this, there were no problems in that place.
Can anyone suggest how to properly process 10k+ lines in a file and send a request to the server for each line? I don't really care about the sequence, the main thing is to process each record.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
G
Gorod_b, 2017-07-11
@Gorod_b

The problem in the end turned out to be not even in asynchrony, although it looked like it, but in speed. The server did not have time to process and answered a little incorrectly)
It was just necessary to add delays.

A
Alexander, 2017-07-08
@boratsagdiev

I usually use the async module ( https://caolan.github.io/async/docs.html) and its mapLimit method for such tasks.
UPD: like this: https://gist.github.com/alcovegan/bb21d9f0760d992b...

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question