Daemon in celery for django: how to configure?

Question

Django

I am using Celery 4, Django 1.11, Python 3.5. I met Celery quite recently, just yesterday I was able to launch the first task with the worker manually enabled.
There was a problem configuring the daemon.
Opened the documentation:
docs.celeryproject.org/en/latest/userguide/daemonizing.html
Chose the option with celeryd, moved this file:
https://github.com/celery/celery/tree/3.1/extra/ge...
to folder etc/init.d/ without changes.
I created a celeryd file and added the text specified in the documentation as a test configuration, and moved it to etc/default/.
And then I don't understand anything. I will try everything in order.
1. What should the default setting look like?
Now I have this:

CELERYD_NODES="irr_parser"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
CELERY_APP="irr_parser"
CELERYD_CHDIR="/opt/Myproject/"
CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
CELERY_CREATE_DIRS=1

irr_parser is the name of the project and also the folder where settings.py and other relying files are stored along with celery.py.
So. CELERY_APP - what to enter here? It says "App instance to use", Google said "App instance to use" - which application? Which has a task for celery? Or is it some other application?
And how to create a celery user? I googled, but did not understand the answer And why CELERYD_CHDIR
="/opt/Myproject/"? I still don't understand why a redirect is needed somewhere (as I understand it, this is the address where to move while working). Why is this needed and will it interfere with saving the received data ( save images in folders) at relative addresses in a django project?
Plus, this item raised questions:
CELERY_BIN="
I walked along the path, I didn’t see any celery there.
2. What should I do after setting up? The documentation says something about django settings and explains the meaning of variables, several pieces of code are attached, but even with a translator I can’t understand their purpose.
CELERYD_SU_ARGS="-l"
What is this? Why is this? What should I do after I make both celeryd files usable?

Answer 1

First of all, I want to draw your attention to this:
If you "choose" between celeryd or celerybeat, then the choice is meaningless: celeryd is a daemon for executing tasks, celerybeat monitors the schedule of tasks, sending them at a given time for execution. So, if you have a need to run tasks on a schedule, you need to configure both daemons.
Now some answers to questions.
This is an instance of the celery.Celery class, here is the path to it and you need to register it, in the format package[.subpackages].module:variable. For example, I have a project/celery.py file that creates an instance like this:

from celery import Celery

app = Celery('project')

And I will enter the value in CELERY_APP project.celery:app
You probably also have a similar code in the project, if you did it according to the documentation
I use one user for the whole project, the one under which django is running (I have a webapp)
I do not use this option, instead I use WorkingDirectory at the systemd level, but the meaning is absolutely the same.
This is not a redirect, it's just an indication of the working folder relative to which your daemon will be launched. With the right settings, the work of the project in celery worker mode should be no different from the work of the project in web mode. At the code level, there shouldn't be any branches on the topic "if I'm a demon, then do this, and if not, do that" (and I don't even know how to do them at all).

CELERY_BIN="/usr/local/bin/celery"
Went along the path, didn't see any celery there.

Well damn, where do you install celery, its executable file will be there. For example, I have all python dependencies, which includes celery, installed in a virtual env, and the path to it is:/home/webapp/Env/project-backend-3.5/bin/celery
After setting - run, test, debug. Everything is as usual.

CELERYD_SU_ARGS="-l"
What is this? Why is this?

I don't know, and probably not. It even says not recommended.
Here are my configs (for systemd), based on configs from the official celery turnip.
celeryd service file (/etc/systemd/system/celery.service):

[Unit]
Description=Celery Service
After=network.target redis.target

[Service]
Type=forking
User=webapp
Group=webapp
EnvironmentFile=-/etc/systemd/celery.conf
WorkingDirectory=/home/webapp/project_dir/project/src
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p ${CELERYD_STATE_DIR}
ExecStartPre=/bin/chown -R ${CELERYD_USER}:${CELERYD_GROUP} ${CELERYD_STATE_DIR}
ExecStart=/bin/sh -c '${CELERY_BIN} multi start \
    ${CELERYD_NODES} \
    -A ${CELERY_APP} \
    --pidfile=${CELERYD_PID_FILE} \
    --logfile=${CELERYD_LOG_FILE} \
    --loglevel=${CELERYD_LOG_LEVEL} \
    ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait \
    ${CELERYD_NODES} \
    --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart \
    ${CELERYD_NODES} \
    -A ${CELERY_APP} \
    --pidfile=${CELERYD_PID_FILE} \
    --logfile=${CELERYD_LOG_FILE} \
    --loglevel=${CELERYD_LOG_LEVEL} \
    ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

celerybeat service file (/etc/systemd/system/celerybeat.service):

[Unit]
Description=CeleryBeat Service
After=network.target redis.target rabbitmq.target

[Service]
Type=simple
User=webapp
Group=webapp
EnvironmentFile=-/etc/systemd/celery.conf
WorkingDirectory=/home/webapp/project_dir/project/src
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p ${CELERYBEAT_STATE_DIR}
ExecStartPre=/bin/chown -R ${CELERYD_USER}:${CELERYD_GROUP} ${CELERYBEAT_STATE_DIR}
ExecStartPre=/bin/rm ${CELERYBEAT_SCHEDULE}
ExecStart=/bin/bash -c '${CELERY_BIN} beat \
    -A ${CELERY_APP} \
    --workdir=${CELERYBEAT_WORKDIR} \
    --pidfile=${CELERYBEAT_PID_FILE} \
    --logfile=${CELERYBEAT_LOG_FILE} \
    --loglevel=${CELERYBEAT_LOG_LEVEL} \
    --schedule=${CELERYBEAT_SCHEDULE}'
ExecStop=/bin/systemctl kill celerybeat.service

[Install]
WantedBy=multi-user.target

celeryd and celerybeat services configuration file (/etc/systemd/celery.conf):

# See
# http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#available-options

# Common Celery Settings
CELERY_BIN="/home/webapp/Env/project-backend-3.5/bin/celery"
CELERYD_USER="webapp"
CELERYD_GROUP="webapp"
CELERY_APP="project.celery:app"

# Common env settings
DJANGO_SETTINGS_MODULE=settings.production
LC_ALL=ru_RU.UTF-8
LC_LANG="ru_RU.UTF-8"
LANG=ru_RU.UTF-8

# Worker settings
CELERYD_NODES="w1 w2 w3 w4 w5 w6"
CELERYD_OPTS="-Q:w1 default --autoscale:w1=8,4 \
-Q:w2 queue2 --autoscale:w2=6,2   \
-Q:w3 queue3 --autoscale:w3=8,2 \
....................
-Q:w6 queue6 --autoscale:w6=6,2       "

CELERYD_MULTI="multi"
CELERYD_STATE_DIR="/var/run/celery"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_LOG_LEVEL="INFO"

# Beat settings
CELERYBEAT_STATE_DIR="/var/run/celerybeat"
CELERYBEAT_PID_FILE="/var/run/celerybeat/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
CELERYBEAT_LOG_LEVEL="INFO"
CELERYBEAT_SCHEDULE="/var/run/celerybeat/schedule"
CELERYBEAT_WORKDIR="/home/webapp/project_dir/project/src"