Answer the question
In order to leave comments, you need to log in
How to organize a double queue in Python?
Good afternoon.
I'm learning Python, how queues work. I look towards Celery.
The point is the following. There is a task to organize processing of incoming messages. Schematically, everything is presented below in Figure 1.
It is necessary to distribute incoming messages among Workers for processing and further transmission to another program. But there is a limitation - only one processing can be performed at a time for a particular devID. Those. if the Worker has not yet processed the past message for some devID, and a new message with the given devID has already arrived, then it should wait for the processing of the past message to complete. In fact, it turns out a double queue: by devID and by Workers.
I do not understand how to do this through Celery. Or maybe I'm digging in the wrong direction at all?
Picture 1.
Answer the question
In order to leave comments, you need to log in
Two queues, one for all messages, the other for those waiting.
The worker:
1) receives a message from the public queue
2) checks for a lock by devID
3) if there is no lock:
3.1) puts a lock on devID
- does the job
- releases the lock on devID
3.2) there is a lock:
- republish the message to the pending queue ONLY if it is from the main queue
4) checks the waiting queue
- if there is a message, then it returns to n2
- otherwise to n1
Something like this ... Well, consider the issue of didlock, when a message in the waiting queue is blocked by a hung worker ...
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question