How to optimize Celery+Redis?

Question

Django

The project (on Django) uses Celery for generating reports, sending messages, periodic tasks, etc. Periodic tasks have their own task_routes, shared/tasks have their own. However, in production, we encountered a problem that a large number of small tasks take up a queue, due to which others are completed with a large delay.
What is the best way to organize a queue, set up Celery / Redis, or what else can be done to make tasks run with less delay?
Celery workers started with concurrency 4, auto_scale 8.32

Answer 1

more workers, more queues.

Answer 2

You need to allocate a separate queue (queues) for priority tasks, and process everything else separately. This way the small ones won't block the important ones.