A
A
Andrey Kobyshev2015-05-07 00:20:46
Django
Andrey Kobyshev, 2015-05-07 00:20:46

How to make a SIMPLE queue in Celery (Python)?

Guys who comprehended the Zen of Celery! Tell me please.
A quick googling on the topic did not give an understanding of how to do this in Celery.
a65a44a3f0294b0fbdbf003c716df6c0.jpg
There are 2 systems. The first is new on python/flask, the second is ancient on ibm/db2 (i400). From the new system, when certain events occur, it is necessary to send single sql queries to the old one (insert/update/delete). They must be executed in a strict sequence (no asynchrony - the previous one has been completed, then and only then do the next one). Moreover, if any error occurs, the task should be placed at the BEGINNING of the queue and try to run again (for example, a network problem or the old system has crashed). Other tasks must wait for this one to complete. It doesn't matter what you have to wait. It's important to keep a strict execution sequence when the bug is fixed.
I use Redis as a broker for Celery.
This process is now working on a self-written solution using the Redis queue mechanism and self-written binding, which monitors execution in one thread and in strict sequence, in case of accidents, endlessly putting the last task at the head of the queue and restarting it (of course, with a delay). But since It was decided to translate all this self-writing under the popular and supported Celery, please tell me the recipe for making this in Celery.
Thank you for your attention!

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
marazmiki, 2015-05-07
@marazmiki

How to execute asynchronous tasks in a sequence is written in the canvas section of the celery documentation. There is also an example (I copy to save time)

>>> from celery import chain
# 2 + 2 + 4 + 8
>>> res = chain(add.s(2, 2), add.s(4), add.s(8))()
>>> res.get()
# или даже так
>>> (add.s(2, 2) | add.s(4) | add.s(8))().get()
16

here add, if you don’t understand, is an example task that is ubiquitous in the documentation. She just adds two numbers. And the dreaded .s() method is just shorthand for .subtask(). In the same documentation about canvas, all this is said in the Signatures section.
As for repetitions of overwhelmed tasks, everything is simple there and the documentation is there. The only thing you need to set up is RESULT_URL. Pay attention, not BROKER_URL, but RESULT, for some reason many confuse

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question