V
V
Vasily Norman2015-01-13 17:00:04
Parsing
Vasily Norman, 2015-01-13 17:00:04

How to correctly implement page loading in X threads in Node.js?

Tell me how to correctly implement page loading in several threads (synchronously) in the implementation on node.js. For example: there is a site with artists, you need to parse it into the database. Let's say there are collected links that provide access to pages with descriptions of artists. There are 200 links in total and you need to load 5 pages at a time (the number may change), parse and write the collected result to the database.
The http module (http.get(URL)) is used to download pages. I feel that in this case it is necessary to use some kind of asynchronous solution like a promise queue, but perhaps some more elegant solution?
Thank you.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
vsvladimir, 2015-01-13
@vsvladimir

Loading in x threads:
stackoverflow.com/questions/15533448/node-js-http-...
But if it is important to wait for x pages to load and then do something with them, and then load the next portion, then the Async module will be convenient .

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question