How to properly optimize the load on Django?

Question

Django

Hello

There is a project on Django with 5000+ users
Within the framework of this project, a large algorithm is executed (about 7 nested loops)
It is calculated for every 30 seconds of the current day, as a result, the lowest function is entered more than 200,000 times. On average, the calculation takes ~3 seconds, with a larger volume it can take 5-10 seconds.
The problem is that when this operation is performed, the server itself starts to hang and wait for another user to receive the result of the calculation.
I want to optimize the prod version as much as possible.

Django runs in a single instance in docker. Works with daphne. The project uses a lot of websocket events (they said that only in it everything with a websocket works like a clock)
I start the container using the command. Nginx proxies all requests to it.

command: sh -c "python manage.py migrate && daphne -b 0.0.0.0 --proxy-headers core.asgi:application"

The server itself has 8 cores, 16 RAM and a 200 ssd.

Maybe it's worth changing the publication or there are some pitfalls that I don't know about. I'd love to hear all the advice!

Answer 1

Perform heavy operations asynchronously, in a separate process, especially if these are cpu-bound tasks. Not a Django user, but I'll assume Django Q will help you here.

Answer 2

First. daphne - in the trash. I gave up on it at the time and moved to gunicorn + uvicorn. Because daphne is not configurable at all. How many worker processes do you think you have? How many streams? How many event loops? Why does he need certificates? So it's a server that can stick out? But it's not like that.
The second - depending on the type of task that is performed "3-10 seconds", you need to either score on it or put it in the queue. If this is IO, then the asynchronous function can last at least an eternity, it will not affect neighboring requests (in fact, of course it will, but very slightly). If it's a CPU, only task queues. Celery for example. But there are also a lot of problems with it, so lately we have been writing self-written django-management commands that listen to the queue.
Third. Junga in general is not very well suited to asynchronous design (and websockets, respectively). Yes, there are clumsy attempts in the form of channels and even django 3 to befriend it with the asinka world, but these are always big compromises with a lot of "buts". Therefore, if there is an opportunity (for example, the project started "yesterday"), then move to something more capable in real asynchrony (fastapi for example)

Answer 3

one)

Itself is calculated for every 30 seconds of the current day, as a result, the entry into the lowest function occurs more than 200,000 times. On average, the calculation takes ~3 seconds, with a larger volume it can take 5-10 seconds.

To answer the request of each user?
2) how many requests per minute? Is there a certain number of requests at which the performance is satisfactory?
3) I would look at the load on each CPU, per disk and database (if any).
It might be worth running multiple instances.