Is there a ready-made framework for organizing a queue pipeline?

Question

Message Queues

Hello!
I ask you to suggest a ready-made framework for organizing a queue pipeline, preferably based on RabbitMQ
. There is a project that processes an array of data in several steps using modules wrapped in docker containers. When the step is completed, the self-written manager creates a queue in RabbitMQ, raises the container (s) and passes them the queue parameters for the connection. After that, the containers process the tasks in the queue. As soon as the tasks in the queue end, the transition to the next step occurs.
Such an architecture significantly slows down the time for the material to pass through the entire processing chain, since there can be quite a large number of tasks in one step, and until all of them are completed, the transition to the next step does not occur. I'm looking for a ready-made framework in which I could set up rules for shifting tasks from one queue to another and deleting them.

Answer 1

www.celeryproject.org