How to run parallel tasks in PHP?

Question

PHP

When the user clicks on the link, a task should be launched.
The average time to complete one task is 12 hours. In each task, several thousand actions are performed with a certain time interval between each action (10-60 seconds). The bottom line is that a user can run several tasks and all of them must be performed in parallel, there can be many such users (10-1000 people).
I dug in the direction of Gearman, but it performs tasks in turn, and it seems to me that it is not rational to create many workers, because I cannot know how many users there will be, when and how many tasks they will run.
If it’s still worth using Gearman and a lot of workers, then tell me how resource-intensive it is?

Answer 1

A tricky task.
Whatever you do, everything will be resource-intensive, but come on, you have 1000 people online who have launched several tasks that perform several thousand actions))
I don’t see a problem in launching many workers. You can use this development of mine to automatically start and stop workers, there is a class from which you can control workers.
https://habrahabr.ru/post/212761/
Algorithm: a task has appeared - we throw it on the queue server - we launch one worker. The task is completed - we stop one worker.
Workers can be located on different servers. You can monitor on which server how many workers are running, and download the free ones.
Thus, the task with the help of Gearman turns into a distributed one.
By itself, Gearman consumes almost no resources. Resources will be consumed by workers, but you can’t get away from this at all - after all, workers solve the problem, no matter how you solve the problem, you will consume the same resources.
More. You don't need multiple Gearmans to distribute tasks. You need one Gearman queuing server that registers workers from different servers. In general, enter the task in more detail somewhere in the personal, I can help in more detail.

Answer 2

I agree with Konstantin Kiyashko
From myself I will add: dig for the keywords "worker pool" --- there are many ready-made semi-finished products for creating workers and worker pools.