G
G
guyasyou2018-07-23 17:25:21
Message Queues
guyasyou, 2018-07-23 17:25:21

How to organize one task queue between different servers?

Hello!
You need to make a web application that works on the API "asynchronously", that is, API requests do not require an immediate response.
Here's how I imagine it right now.
5b55e2742b948748187406.png
Servers 1, 2, 3 are doing the job of processing requests ... These servers only read MySQL, they don't need to write anything.
The request to the API is sent to one of the servers, which puts the task in the queue. And the workers take an available task and perform it.
So here's how to organize a single queue between servers so that:

  • one task was taken by only one worker
  • a task queued by one server could be executed by any other
  • the failure of one server did not affect the work of others in any way

For queuing, I considered Redis. But I understand that it works on the master-slave principle, and I need the servers to be self-sufficient. Can you suggest something else?
Or is the architecture wrong in general?
Speed ​​is not so important, the main thing is to be able to easily scale and reliability.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Doc44, 2018-07-23
@Doc44

There are specialized queue servers.
Why stick Redis everywhere and steer the queue under it manually.
RabbitMQ, for example.

R
RidgeA, 2018-07-23
@RidgeA

I don't quite understand the task queue at the bottom of the diagram.
There is a web interface that will accept the request and queue the task.
There are N workers (+ in RabbitMQ that workers can be raised and lowered dynamically, depending on the load) that are subscribed to the queue. RabbitMQ round-robin distributes messages to workers. You can adjust the number of messages that the worker can process at a time and use it in the message confirmation mode (if the worker did not say that ok, the message will return to the queue)
After processing the task, the worker makes changes to the database.
The web interface can freely read the database and find out if the task has been processed or not.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question