Answer the question
In order to leave comments, you need to log in
How to correctly implement page loading in X threads in Node.js?
Tell me how to correctly implement page loading in several threads (synchronously) in the implementation on node.js. For example: there is a site with artists, you need to parse it into the database. Let's say there are collected links that provide access to pages with descriptions of artists. There are 200 links in total and you need to load 5 pages at a time (the number may change), parse and write the collected result to the database.
The http module (http.get(URL)) is used to download pages. I feel that in this case it is necessary to use some kind of asynchronous solution like a promise queue, but perhaps some more elegant solution?
Thank you.
Answer the question
In order to leave comments, you need to log in
Loading in x threads:
stackoverflow.com/questions/15533448/node-js-http-...
But if it is important to wait for x pages to load and then do something with them, and then load the next portion, then the Async module will be convenient .
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question