How to speed up or run multi-threaded pupeteer?

Question

JavaScript

There is a parser on puppeteer, they need to parse 60 thousand pages, in time one page takes 3.6 seconds, if converted to hours, then all this stuff will take 61 hours. Is there an option to speed up puppeteer or run it multi-threaded so as not to wait so long?

Answer 1

1) Divide 60,000 pages into portions that are multiples of the number of instances. Each instance to feed its own bunch. Get an acceleration multiple of the number of instances.
2) Queue 60000 pages (rabbitmq, redis, db). Scripts take url to work from the queue. There are still enough resources to launch new workers. https://www.rabbitmq.com/getstarted.html
Also may be useful https://habr.com/ru/post/413547/