Why is the database crashing and the entire Django server behind it?

Question

PostgreSQL

I apologize in advance for the spam tags, but I don't know where to dig and what causes the error.

After running docker-compose up, 4 containers rise:
redis - I use only for django-channels
django - I start asgi, through uvicorn
postgesql
nginx

It normally connects to websocket and works, but after some time (about 100 minutes, it happened three times) nothing works anymore.

Attempts to enter the django admin end with this error (the screen was found on the network, but the error is exactly the same)

could not fork new process for connection: Resource temporarily unavailable django

:


I can't connect to the server via ssh, error:

kex_exchange_identification: Connection closed by remote host

If at the time the problem occurs, you are in the console on the server and call something, then this error:

-bash: fork: retry: Resource temporarily unavailable

I solve it by restarting the server from the vps control panel.

I don’t know why such an error occurs, on the network you can find information about descriptors, the number of simultaneous connections in the database and other reasons, but I don’t want to mindlessly execute the suggested commands in the console.

Do you have any clues and hints where to look for the cause of the problem? Is the database only to blame?
If so, what settings should I consider?
In the logs that come to mind, too, did not find anything.
Can a websocket connection block the database somehow? now quite often you need to connect / disconnect from the socket.
Although I see in the console output that there is both connect and disconnect.

Answer 1

Launched docker statsand began to use apishka as usual.
I noticed that every websocket connection I made increased the PIDS value of the above command for the Django container.
The description of the command in the docker documentation states that a large value of this column indicates that extra threads are being created inside the container.
Removed extra lines of govnokoda and now everything works.

Answer 2

Sounds like the server is running out of memory. It is worth monitoring its use, find where the leak is and fix it. Yes, and it would be nice to configure the server so that it does not allow this.

Answer 3

docker-compose 2.3 still allowed resource limits. Use it and set limits on the amount of available memory for containers.
Set up monitoring (node_exporter, Prometheus, Alertmanager, Grafana) see what happens to the server - here are the instructions: 1 CPU 1 GB - and I want monitoring, like the big uncles
PS
If you collected images based on Alpine, be careful, there is a surrogate instead of libc called musl which is very limited and can add various special effects similar to what you described.