Answer the question
In order to leave comments, you need to log in
How to organize one task queue between different servers?
Hello!
You need to make a web application that works on the API "asynchronously", that is, API requests do not require an immediate response.
Here's how I imagine it right now.
Servers 1, 2, 3 are doing the job of processing requests ... These servers only read MySQL, they don't need to write anything.
The request to the API is sent to one of the servers, which puts the task in the queue. And the workers take an available task and perform it.
So here's how to organize a single queue between servers so that:
Answer the question
In order to leave comments, you need to log in
There are specialized queue servers.
Why stick Redis everywhere and steer the queue under it manually.
RabbitMQ, for example.
I don't quite understand the task queue at the bottom of the diagram.
There is a web interface that will accept the request and queue the task.
There are N workers (+ in RabbitMQ that workers can be raised and lowered dynamically, depending on the load) that are subscribed to the queue. RabbitMQ round-robin distributes messages to workers. You can adjust the number of messages that the worker can process at a time and use it in the message confirmation mode (if the worker did not say that ok, the message will return to the queue)
After processing the task, the worker makes changes to the database.
The web interface can freely read the database and find out if the task has been processed or not.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question