How to process a large amount of information at once using Cron?

Question

linux

Good afternoon, ladies and gentlemen! The task is as follows: to take certain information from a huge number of links and write it to the database (every 2 hours by Cron).
If you write them in one PHP file, then it will take a lot of time and with an unforeseen error, everything will crash.
The first idea that came to my mind is to create a lot of PHP files and run them with the sh script, but there are a very large number of these links, and as a result, there can be more than 1000 files.
How to implement this task correctly?

Answer 1

I haven't said this for a long time, but! Queue to help you. RabbitMQ, for example, as a broker. Once every 2 hours, you throw tasks into it with one script, and on the other side of the queue, put as many identical handlers as you like. The more, the faster the processing.

Answer 2

Here is an option
https://andy-blog.ru/ispolzovanie-curl_multi_exec-v-php