Is there an analogue of Cron with the distribution of tasks between servers?

P

Philipp2014-03-02 19:59:58

PHP

Philipp, 2014-03-02 19:59:58

I'm trying to find a scheduler with task distribution inside a server pool.
For example, I have 10 tasks that must run every day at 9.00 am and 4 servers. At 09:00, tasks evenly or, if the system allows you to add a coefficient of task complexity and server performance, depending on the load, distributes tasks with the formation of a queue for each server. But during distribution, execution control is used, and if the task is not completed within a certain interval, then it is redistributed between other servers. Those. system scalability and fault tolerance should be the default.
I saw RabbitMQ & Gearman, they can help with something, but not as a scheduler. Now I have my own simple bicycle, but it is applicable only to one server, and I need scalability, because. it is planned to process about 1000 tasks per minute.
Technically I need to run a php script with a parameter. Something like workerClient.php -task36762. Inside the script, there should be something like taking over and confirming that the task is completed with such and such a result. Ideally, if the system will use MongoDB.

Reply

Answer the question

In order to leave comments, you need to log in

4 answer(s)

Y

Yuri Shikanov, 2014-03-03
@zoonman

Celery

I

Ivan Starkov, 2014-03-02
@icelaba

What is stopping you from keeping open processes on servers hanging on rabbitMQ queues using ack:true, prefetchCount:N ,
and managing tasks from one machine - running cron ohm the control process,
Rabbit will solve such your tasks as execution control
(if the server falls off or the process crashes then the task will be rescheduled automatically),
about rescheduling by timeout - if the server is running and has not fallen (then what's the point - the task will also hang another server)
Equal load tasks, etc.
Rabbit is a very convenient thing, and it will work great for 1000 tasks per minute.

A

Anatoly Evladov, 2014-03-02
@Visteras

Um... I didn't understand a bit - if you need to pull only php scripts - then what's the problem with pulling them all from one computer with the most common cron?
Or I misunderstood what you need?

V

Vlad Zhivotnev, 2014-03-03
@inkvizitor68sl

https://github.com/noxiouz/python-flock + regular cron.
Before executing the task, a zookeeper lock is taken, so that the neighboring machine will no longer execute the task (of course, you need to somehow mark in the database that the task has been completed - so that at the end of it the other machine does not take the lock for this task again).