How to fix memory problems?

Question

linux

On a non-weak, in general, server, any command is issued

-bash: fork: Cannot allocate memory

in a rare moment of enlightenment got top

top - 17:52:36 up 50 days,  6:13,  1 user,  load average: 1.01, 1.13, 1.03
Tasks: 1868 total,   2 running, 1866 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us, 33.9 sy,  0.0 ni, 64.4 id,  0.8 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem :  2052332 total,    59784 free,  1930936 used,    61612 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    12570 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   37 root      20   0       0      0      0 R  52.4  0.0 126:43.98 kswapd0
 5553 mysql     20   0  691788 147584      0 S   6.9  7.2  31:10.36 mysqld
15691 root      20   0   48052   3160    556 R   4.4  0.2   0:00.43 top
  455 root      20   0  572420  16564      0 S   1.6  0.8  55:10.43 containerd
26602 root      20   0  298640  15884      0 S   1.6  0.8   6:07.22 php
 5456 root      20   0  725056  18412      0 S   1.3  0.9   7:01.30 core
  736 root      20   0  417512 141040      0 S   0.9  6.9 233:22.80 fail2ban-s+
 5667 root      20   0 1363264 113900      0 S   0.9  5.5  26:30.04 core
    7 root      20   0       0      0      0 S   0.3  0.0  31:45.10 rcu_sched
  142 root      20   0       0      0      0 S   0.3  0.0   3:11.06 jbd2/vda5-8
  854 ntp       20   0  102104    584      0 S   0.3  0.0   4:06.90 ntpd
15602 root      20   0       0      0      0 S   0.3  0.0   0:00.10 kworker/u4+
    1 root      20   0  270292  67236      0 S   0.0  3.3   1:17.53 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.36 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   2:17.78 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh

I personally do not understand - there seems to be no swap in the system, but the kswapd0 process eats up half of the CPU .
There are also a lot of sleeping tasks. I suspect that they originated like this:
A php process is running on the server, which I would like to have constantly running. There is set_time_limit(0); But, it is clear that this does not guarantee against various force majeure. Therefore, this process is launched by cron every minute and first of all it looks to see if there is a duplicate in ps axu. If there is, then it stops working. Probably, at some point the system turned out to be heavily loaded and the process, although it started, could not even check for a double. After 1 minute it happened again and so on.
So I have 2 questions.
1. Will creating a swap file help in the described situation? Or does he already exist?
2. How to organize that the necessary php script is guaranteed to be executed, and in the event of a crash - restarted?
Thank you.

Answer 1

The output of top can be sorted by memory by pressing inside top Shift/M.
Better yet, run them with cron every five minutes
top -b -o +%MEM | head -20 >> /tmp/top-memory.out
This way we will quickly understand who is eating memory.
Another option is to count PHP processes:
ps -ef | grep php | wc -l
Perhaps your script is not good at killing extra instances
Not yet:
KiB Swap: 0 total,
Can be created, but most likely it will just delay the problem.

Answer 2

Known issue of the Linux kernel, described e.g. here .
In a nutshell: with a lack of RAM, and you have it left, judging by top, 59 MB out of 2 GB (2%), the system starts to respond extremely poorly to requests. Therefore, you should not look for the reason in the CPU load.
You need to work on optimizing the use of RAM or increasing it. Enabling swap if it helps, then not for long.
The error "Cannot allocate memory" is about this - the memory cannot be allocated, simply because there is no free memory.
The processes listed in your top output don't even come close to taking up that much memory -- look for the problem among your sleeping processes.