How to organize automatic parsing?

Question

Java

I have a Java web application that parses html. But it parses only 1 site per visit, how can I make it so that I would just take a list of urls, tags that will be parsed by the end of work?
As I understand it, by and large, you can simply make a collection with urls and iterate over this collection each time opening the url and scavenging what you need

Answer 1

Don't reinvent the wheel, use standard web scraping frameworks, the so-called crawlers.
For example - https://github.com/yasserg/crawler4j or here https://github.com/DigitalPebble/storm-crawler
, into as many threads as needed.

Answer 2

Clustering / Scaling well, as the streams have already suggested