How to parallelize a process?

Question

Java

I came up with a problem for myself: there is a file with 100+K links, which are periodically updated, you need to send for each request, get a response and process it somehow: parse it. For the fastest solution to such a problem, I googled the thread pool, maybe there is a more beautiful way to solve such a problem?

Answer 1

there is a file with 100+K links, which are periodically updated

Use a queue. For example, https://ru.wikipedia.org/wiki/RabbitMQ
The most efficient way is in bash (curl/wget, grep, sed, etc.). But it is possible in any other PL, including Java.

Answer 2

Use asynchronous requests, such as Go, where it is out of the box, or Python with asyncio, or any language that you know

Answer 3

There is Spring Batch for these purposes (for bulk processing) and some kind of queue broker, you can RabbitMq. Here is an example of spring batch integration