F
F
Fedor Ananin2015-06-15 16:06:25
PHP
Fedor Ananin, 2015-06-15 16:06:25

How to make a beautiful scheduler for PHP script?

Good localtime!
Friends, there is such a situation:
1) a certain API with a limit of 3 requests per second
2) a large queue of requests to the API
It turns out that the script can execute more than three requests per second without any problems ...
The server is running CentOS, the service is written in PHP without frameworks.
Googled in the direction of CRON ... There you can call the script a maximum of once a minute ... So my task queue will quickly line up for months. Alternatively, you can do this:

<?
$i=0; $max=45;
while ($i<$max){
  //Начало запроса к АПИ

  //Конец запроса к АПИ
$i++;
sleep(1);
}
?>

And pull this script every minute with a cron. The script will execute 45 requests per minute, so there will be time just in case and there will even be a margin for the number of requests per second. The method is scalable, you can put two API requests into the loop and the total number of executions will be already 90 per minute.
The script will have to make queries to the database. I see the general scheme as follows:
1) request for all rows from the database, limit 45
2) start of the cycle
3) request to the API
4) request to the database with updating information ("task No. x completed" or "task No. x completed with such and such an error ")
5) $i++
6) end of the cycle, everyone is happy
In the future, it is planned to integrate other APIs into the service, that is, very soon there will be 4-5 such worker scripts.
Several questions:
1) how beautiful does this solution look?
2) Are there any better solutions?
3) there will be moments when the task queue will be empty, but cron will still pull the scripts, is it scary? (4 scripts, 4 queries to the database per minute...)
4) Will this solution heavily load the server? (e.g. $5 droplet on DigitalOcean?)
Many thanks in advance for all the replies!

Answer the question

In order to leave comments, you need to log in

2 answer(s)
H
He11ion, 2015-06-15
@sarathorn

A well-posed question is already half the answer. Cron is not designed for such work, you are right, it is for running a script once in n-time. For your task, you should use Gearman / Beanstalk / something else to your taste, it will be easy to balance the load and manage the number of workers there.
On loading - strongly depends on what calculations/volumes of the data, at once not to tell.

S
Semyon Dubina, 2015-06-15
@sam002

Stop! Here is something that goes wrong (it will take more than a second to process three requests or the php thread will wait a long time to execute) and get a queue block. It is necessary to limit the time with the rejection of overdue tasks. I recommend to cross php.net/manual/ru/class.splqueue.php with an asynchronous daemon (from fresh but untested: https://github.com/walkor/workerman-queue ).
If you want it simpler, then translate the limits to "requests per minute" and then you can process from the cron, as it will be possible to kill dead queue handlers from the startup script.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question