T
T
topuserman2020-02-18 10:53:26
PHP
topuserman, 2020-02-18 10:53:26

Should I migrate from cron to the queue?

Hello everyone, we have 2-3 tasks on the crown, each one checks some data in its interval, and performs heavy background work.
For example, working with a catalog, indexes, or importing data if an upload file has appeared.

At the moment, we are already using rabbitmq, and a dozen consumers for other tasks (for real-time integration with the supplier).

Question: is it worth being afraid to produce consumers? How heavily do they load the system when they just listen to the queue?

Is it worth transferring the execution of tasks by cron, to the queue? Create a queue of tasks, and one consumer who, when a task appears, will launch it.

The fact is that cron runs only 4-5 tasks per day, but checks for their presence every 5 minutes, because. they can appear at any moment.
What scares me is that in the variant with queues, the consumer 95% of his time will just hang and listen to the queue, because. messages fall 4-5 times a day.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
Stanislav, 2020-02-27
@stanislav-belichenko

About the load:
No, nothing critical will happen, unless you work with servers from the 80s. Specifically, the rabbit holds huge loads perfectly.
About expediency:
Firstly, they write correctly above, while it works - do not touch it.
Secondly, again, they correctly write above that the idea is doubtful for such a number of tasks.
Thirdly, queues are usually something eventful. A client came to your site, uploaded his photo, you gave birth to a message in the queue on the backend of the site that the photo should be checked by support, this event was received by the support CRM system and they performed some actions. And in the model described in the question, who will give rise to such events? Some kind of timer? He will again most likely be on the crown, what's the point of building a garden then with a rabbit? If not on a timer, but on some event that occurs in the application at the right time - well, ok. But most likely, it will still be different.

K
Karpion, 2020-02-18
@Karpion

Heavy work is best done in sequence, not in parallel. Although it depends on the number of processor cores and on the number of disks; this is a non-trivial optimization problem.
I didn't understand what "checks for tasks every 5 minutes" means. What exactly is being checked?
If you need to run programs quickly, then you need to make sure that the "source of tasks" (the one who creates the need to run the task) starts the task himself. There are many mechanisms, I can discuss.

R
Roman Mirilaczvili, 2020-02-18
@2ord

You can make this option for launching tasks:
The Task Scheduler launches a program that checks some conditions, and when a decision is received to complete the task, it sends an instruction to the queue with the arguments necessary for execution, and then the program immediately ends. With such a scheme, the program process will not hang for a long time, and consumers will process tasks asynchronously and one by one.
But for such a small number of tasks, the idea is doubtful.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question