Job queue in PHP/MySQL. Am I writing another crutch?

A

Alexander2012-10-22 16:50:18

PHP

Alexander, 2012-10-22 16:50:18

In PHP, I understand very mediocre, but once I wrote a system for collecting statistics that was in demand by a narrow circle of people. Now the script that collects data runs from 45 minutes to more than an hour, and now it's time to scale in time.
I want to write a job queue in PHP, but not bother with RabbitMQ and other MQ, but store a list of jobs in MySQL. There is very little information on Google, mostly they offer ready-made solutions. But I want to figure out what's what.
What I want to do:
Several times a day, a script is launched that fills the job queue (urls).
Once every 1-3 seconds, a worker is launched, which selects a task (url) for itself and starts processing it (downloading and putting it in the database). If there are no tasks, then it dies, if there are, then it performs.
Questions:
How to run a worker, what to do so that the workers do not turn into zombies and do not multiply unnecessarily?
How do they do it correctly and competently?

Reply

Answer the question

In order to leave comments, you need to log in

3 answer(s)

E

egorinsk, 2012-10-22
@Alesot

The easiest way is to run the “supervisor” by cron. The overseer saves its PID to a file, memcache or database, and if it sees that another overseer is already running, it silently dies.
The overseer must be extremely simple so as not to fall. You can detect the fall of the overseer by the sign that there is a file with a PID, but there is no process with such a PID, which means that the previous overseer has suddenly terminated. This can be logged.
If the overseer has started and there are no other overseers, it takes tasks and launches one or more worker scripts (worker), distributes a task to each. If one of the slaves completed successfully, we mark the task as completed, if it died, it logs this and launches a new one, if the task has not been completed after N attempts, it is marked as impossible and is no longer executed.

V

Vampiro, 2012-10-22
@Vampiro

Google phpdeamon. I think a single-threaded daemon is enough here. There is a task - we are working. No - sleep for 10/60/600 sec. The launch of the daemon can be tied to cron.
Competently - do not use the "crutch" pattern, but use a ready-made solution, understanding it along the way :)

Z

zarincheg, 2012-10-23
@zarincheg

Use Gearman. It is extremely easy to use and convenient. Install an extension for PHP, write your workers and that's it. Can store queues in MySQL.