M
M
Max Maximov2018-05-16 09:01:34
.NET
Max Maximov, 2018-05-16 09:01:34

How to set up a classic queue in RabbitMQ, or without it?

Hey!
I have been thinking about the architecture of one project for a long time, but I ran into those. questions in terms of implementation.
Description: There are 100k client computers in, say, 50 internet clubs around the country. After each session of work on these computers, they must connect to send a report from the main server No. 2, which in turn connects to the main server No. 1 in order to transmit the report received from the client, receive a response and transmit again to the server No. 2, and that -- to the client. The time for the client to receive a response from server No. 2 is 10 seconds, if no response is received during this time, the connection is terminated.
There are also 2 types of work: online and offline.
Online:everything works in the order of the classical queue, each report is processed by date. The client started and ended the session -> opened a connection with server #2 to send a report with a 10 second wait -> server #2 received and wrote to the queue -> reports are taken from the queue and sent to server #1 for verification and recording -> server # 1 receives and sends a response to server #2 -> server #2 sends a response about the successful completion of the operation to the client, which waits 10 seconds.
Offline:It happens that server number 1 can fall off at any time. And then the client does not receive a response within 10 seconds, then he says that now he is working offline. But it still continues to send reports to server #2. Server #2 also accepts reports, but writes them to the offline queue, which should also be processed as soon as server #1 starts up. And other reports that come from the client should also be expected. Because server #1 accepts everything sequentially.
Question: how to properly organize such an architecture and on what?
So far I have come to use RabbitMQ and microservices. But I think that it will be bad manners for each client (100k computers) to create their own queue, and we will have two queues: online and offline. The online queue will not be processed until the offline queue ends.But for each computer, everything must be processed sequentially, by date: it turns out such a "classic" Soviet queue. But in parallel with several computers: 100k computers are processed in parallel, but reports from each of them are sequential. Based on the calculations: 100k clients and 2 queues for each = 100 * 2 = 200 queues, 2 queues for each computer.
How to do it right here? Ideas, advice?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
Sergey Tikhonov, 2018-05-16
@tumbler

reports are taken from the queue and sent to server No. 1 for verification and recording

Yeah, this is the place where it is decided whether server number 1 is offline or online.
And this is the main requirement.
Option number 1.
One common queue, one worker. If server #1 is offline, the worker simply stops working until the server comes back. For the current message, you need to perform amqp reject, then it will return to the front of the queue.
The order of messages is observed, but there is a single point of failure, it is also a performance bottleneck (however, it depends on the speed of report processing)
Option No. 2.
One common queue, many workers. The scheme is the same, the server is offline - stop processing. Online - renewal. The performance bottleneck and the point of failure disappear, but two consecutive reports from the same computer can get to neighboring workers at the same time, i.e. the requirement about "sequentially" may not be met.
Option number 3.
HashRing and other hashing algorithms that allow you to reduce the number of queues and hang your own worker on each queue. The need for manual load balancing, points of failure on separate queues.
Something like this. I did not figure out how to force several workers to process one queue while maintaining the order of messages.

E
eRKa, 2018-05-16
@kttotto

It is not entirely clear why there is a queue on each computer? In fact, the queue is needed only on computer No. 2. All clients send a report to it, reports are placed in a queue, computer No. 1 picks them up from this queue. And the queue is needed for this, so as not to depend on whether computer No. 1 is online. All messages will lie in the queue until the first one picks them up. If during some time the message was not taken, the client is given a response that the first one is not available and the client sends another message after a timeout.
Each client has its own identifier, each report has its own identifier, and if there are several messages from one client and with one report identifier in the queue, then the last one by date is taken into account. Well, in principle, that's all.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question