How to correctly compose the logic of an asynchronous program?

Question

Node.js

There is a program that parses data from some site, there are several pages of data on that site, I need to get them all, I can’t figure out how to organize the request queue.
I need to make a request to the site, get a link to the next page (I can’t generate it and don’t know in advance) and then, based on whether it exists or not, make another request and repeat the previous steps. I can’t figure out how to do it asynchronously
, I don’t know the exact number of pages

Answer 1

var request = require('request');


fetchData('site.com').then(function (data) {
  console.log(data);
});


function fetchData(url) {
  return fetch({ method: 'GET', url: url }).then(function (body) {
    var link = findLink(body);

    return link ? fetchData(link) : body;
  });
}

function fetch(options) {
  return new Promise((resolve, reject) => {
    request(options, (error, response, body) => {
      if (error) {
        reject(error);
      } else {
        resolve(body);
      }
    });
  });
}

function findLink(body) {
  // return parsed link or null
}