What is the best way to process a PHP script with an execution time of 5-10 minutes?

Question

PHP

Good afternoon, tell me some advice about optimizing the work of long scripts.
The essence of the problem.
Inside the framework, there is a script that automatically updates some user data (downloads a csv file via API, parses and writes to a table for each user).
The system assumes that for someone this file will be 5,000 lines in size, and for someone it will be 20,000 lines.
Accordingly, when there are 10+ such files, then the script is not embedded in 30 seconds, and worse, it can even break off right in the process of work.
Need advice on optimizing this problem. Cron does not help now, because when it breaks, you have to return to the same file again, and it again "does not fit" in 30 seconds.
What solutions can be in principle?
Do I need to use a queue server, will this solve the problem?
Or do I need to work with the PCNTL library and restart the script?
PS Another factor is that the client himself indicates the update time, for example 17:00. But if there are 100 files at 17:00, then with cron now it can receive an update at 18:00 or not at all

Answer 1

run similar tasks in console mode: there is no time limit in cli.

Answer 2

If the execution of the process can break at any time and you need to guarantee it, then I would use queues. Pack part of your script into a console command that you put on cron (the part that is responsible for selecting users for processing). The second part, which is responsible for downloading the file via api and updating the user, is wrapped in a consumer next in line. This way you can guarantee that all your users will be processed, and for those that did not complete without errors (they will return to the queue), you can add a fix for their correct processing.
Put all your workers (consumers) on supervisord. So that one consumer does not strain over your heap - specify the number of processes in the supervisor config.
To solve the problem over time, you can do it in 2 ways:
1. During the selection on the console command, send to the queue only those users who must now be updated
2. For each message, put down the time field. Check this time at the konusmer, if it's too early, you throw it back in line