M
M
Maxim Zubenko2019-08-04 07:45:05
Django
Maxim Zubenko, 2019-08-04 07:45:05

How to set celery to different queues for different Django projects on the same server?

So there is VDS.
It has two different Djangos with their own virtual environment. In different places.
Each has its own Celery.
RabbitMQ is also installed in the system (so to speak, common to all) and the whole thing is launched by SuperVisor.
Each Django has its own different tasks (task), but one of them in each Django has the same functionality. Namely, the creation of a PDF file. Naturally, when setting up a task, each of these "pdf-tasks" has its own way to save the file.
So all the tasks are lined up in one queue and the following is obtained. I request to do PDF for one project (well, invoice) all the time. And once the pdf-file appears in the folder of the desired project, and the second time in the folder of another. And so in succession.
one-celery.conf (for supervisor)

[program:one-celery]
command=/home/iuser/djangoprojects/one/v_one/bin/celery worker -A one --loglevel=INFO
directory=/home/iuser/djangoprojects/one/
user=iuser
numprocs=1
stdout_logfile=/home/iuser/djangoprojects/logs/celery.log
stderr_logfile=/home/iuser/djangoprojects/logs/celery-err.log
autostart=true
autorestart=true
startsecs=10

stopwaitsecs = 600

stopasgroup=true

priority=1000

celery.py (in the first project)
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'one.settings.prod')

app = Celery('one')
app.config_from_object('django.conf:settings', namespace='CELERY_ONE')
app.autodiscover_tasks()

I almost forgot, there is one more line in settings -> prod.py
CELERY_BROKER_URL = 'amqp://localhost'
for the second project, the settings are exactly the same, only two is written instead of one.
What to change (add) in the above files so that each project has its own task queue?
Sincerely, Maxim

Answer the question

In order to leave comments, you need to log in

2 answer(s)
M
Maxim Zubenko, 2019-08-05
@JawsIk

So, having tried a bunch of options, I got only one. It’s hard for me to say why it is, but maybe it has something to do with the versions of Celery itself, because all the solutions older than 2018 just didn’t work for me. In the end, I combined a couple of solutions (found on stackoverflow) that were proposed in 2018 and 2019 and finally I got it.
So the settings.py file. In general, he was not involved in any way. In it (if you look above) there was only a broker setting. It migrated to the celery.py file
celery.py now looks like this:

from kombu import Queue, Exchange, binding
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'one.settings')

# default_exchange = Exchange('default', type='direct')
one_exchange = Exchange('one_go', type='direct')

app = Celery('one', broker='amqp://localhost')
app.conf.update(
        task_queues= (
            Queue('one_go', [binding(one_exchange, routing_key='one_go')]),
        )
)

app.conf.update(
        task_routes= {
            'plugin.email.tasks.send_email': {'queue': 'one_go'},
            'plugin.print.tasks.create_pdf_task': {'queue': 'one_go'},
            'plugin.print.tasks.create_pdf_from_site_task': {'queue': 'one_go'},
        }
)
app.autodiscover_tasks()

well, the launch command located in the
one-celery.conf file (for the supervisor) , the beginning of which now looks like this:
[program:one-celery]
command=/home/iuser/djangoprojects/one/v_one/bin/celery worker -A one -l info -Q one_go

Perhaps something is superfluous here, maybe the code is redundant, but it works exactly the way I need it. In the second "Janga" everything is the same, only instead of the conditional "one" the conditional "two" is written everywhere. I've been tormented by the task since Friday. As they say, "not even three days have passed." Thanks to all.
ps I don't even know why it doesn't work for me through settings like CELERY_QUEUES or CELERY_ROUTES. Perhaps these parameters have been abolished for the sake of those designs that I wrote in this message above.

T
terentjew-alexey, 2019-08-23
@terentjew-alexey

RabbitMQ has the ability to set up virtual hosts.
https://www.rabbitmq.com/vhosts.html
Accordingly, in your projects you raise a connection to one rabbitmq instance, but to different virtual hosts.

# settings.py
CELERY_BROKER_URL = 'amqp://user:[email protected]:5672/my_first_host'

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question