How to allocate reserved RAM for SSH and/or console?

Question

linux

Hello.

The problem is the following. There is a server on linux/debian. Let's say he runs out of RAM, then the swap ends and the server ends there. I would like to connect to it via SSH in such difficult moments, but as you understand, the server is in full swap and there is nowhere to take memory from, respectively, it does not connect via SSH, even IP-KVM does not help - the result is a hardware restart, RAID resynchronization and hemorrhoids.

Question. Is there a possibility / crutches to allocate reserved memory for certain critical parts of the system?

PS: you don’t need to give advice like “it is necessary to correctly calculate the maximum allocated resources” (execution time of scripts, allocated RAM, maximum number of connections) - this has already been done by itself. Now I am considering hypothetical options that I would like to close for every fireman.

Thank you for your attention!

Answer 1

I recommend getting acquainted with SysRq , this will help you properly reboot / shut down the collapsing server.
Most often, the server stops responding. OOM comes and shoots itself in the foot, destroying the init process, which pulls all the other processes along with it, incl. sshd, console and services.
So if the server stopped responding - 99% that it currently has just a car of memory)
1. Make more swap.
2. Monitor the state of the zabbix / ngaios server and when the threshold is crossed, conditionally, 30% swap used - go and check what is happening.
3. Tighten an automator (dangerously) restarting / destroying processes (monit) if programs have the property of "leaking" over time.
4. To clamp through ulimit users on behalf of which the processes devouring memory work.

Answer 2

I usually write a simple script that checks the use of the swap, if more than half uses it stupidly kills, say, a snickering Apache and starts it again, there is no better solution yet.