V
V
Vladislav Ross2014-02-12 11:25:48
Distributed Computing
Vladislav Ross, 2014-02-12 11:25:48

How to organize message queues (like RabbitMQ) with processing speed limit?

I need to organize parallel processing of jobs on several machines. Each task is a call to one of 10 external systems that it is undesirable to pull more than a given interval. Each system has its own: approximately from 0 to 5000 ms.
I need a cunning queuing mechanism so that on the one hand any handler can receive any task, and on the other hand, tasks are processed no faster than the minimum interval.
I’m only reading about RabbitMQ so far, but it seems that he doesn’t know how.
Of course, you can track this on the producer's side and not add tasks faster than necessary, but I would like to just throw tasks into the queue, and let it think about it herself.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexey Kiselev, 2014-02-12
@alexeykiselev

It is necessary to limit the rate of consumption of messages from the queue. If external systems need to rest, then the processor consuming tasks from the queue must ensure this condition is met.
If the consumers are already written and you do not have their code, you can put them on separate queues and write a simple router that transfers messages from the general job queue to the job queue for the external system with the desired delay.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question