OS does not release memory. What could be?

Question

linux

Hello.
There are about 10 servers, partly CloudLinux 6 is installed (this is an OS based on CentOS 6), partly CentOS 6. Faced the problem of allocating and freeing memory.
Initially, I noticed that MySQL servers (there are versions of MariaDB 10.2.17 and Percona 5.6.39, depending on the server) consume more memory than they are allocated. For example, mysqltuner.pl might have the following output (this is a test server, buffers are set to minimum):

-------- Storage Engine Statistics --------------------------------------- --------------------------
[-] Data in MyISAM tables: 284.6M (Tables: 8400)
[-] Data in InnoDB tables: 2.1G (Tables: 9416)
[-] Data in MEMORY tables: 0B (Tables: 62)
-------- Performance Metrics ---------------------------------------- -------------------------------
[-] Up for: 2m 34s (7K q [47.175 qps], 696 conn, TX: 23M, RX: 18M)
[-] Physical Memory: 7.7G
[-] Max MySQL memory: <b>562.8M</b>
[-] Other process memory: 2.2G
[-] Total buffers: 184.0M global + 2.9M per thread (100 max threads)

And the database server can eat up 2 gigabytes (namely RES, not VIRT memory) after a couple of days of work (or you can just run mysqltuner.pl several times and usually the consumption is already above the specified limit). flush tables, caches do not release memory.
There are two solutions to this problem - this is setting MALLOC_ARENA_MAX to a value from 1 to 4 (initially this value for a server with 8 cores is 64), in this situation MySQL starts to consume memory correctly. You can also set malloc_check_=3(also helps, this variable puts glibc into old threading mode). The systems have glibc 2.12 version, for some reason RHEL compiles it with the --enable-experimental-malloc parameter, which creates these problems (changing malloc_check_ just disables this mode). Second solution, use jemalloc (specified by malloc-lib in my.cnf file).
I found similar problems on the net, but often it describes the uncontrolled growth of VIRT memory on glibc 2.12, but not RES. The only similar issue entry is here https://jira.mariadb.org/browse/MDEV-15344.and also found on the site bugs.mysql.com, but did not save the link. If you look at pmap -x, then you can see a lot of allocated anon blocks of 10 and 64 mb. Another interesting point is that when starting the server through the valgrind --leak-check=full --leak-resolution=med utility, memory consumption is normal.
This problem is not only with the database server , there is an interesting point with the nginx web server. At startup, it consumes 1.4 gigabytes (RES) of memory, if you perform a soft reboot (graceful), then after the "rebirth" of processes, new ones begin to consume 3 gigabytes (RES) of memory. Switching to jemalloc also fixes this behavior.
For diagnostics, I installed two new servers with identical resources and operating systems, copied the database and web servers, the problem could not be reproduced. I began to compare all the sysctl parameters, the kernel, there were no differences. I checked the files of all packages using rpm-Va, and then did yum reinstall * (which reinstalled all packages), but the problem did not disappear on the test server and did not appear on two specially installed servers. I tried to refuse and update glibc, there is no result either.
What could it be, where to dig? Why does the same software behave differently?

Answer 1

Initially noticed that MySQL servers.... consume more memory than they are allocated.

During memory allocation, the operating system is up to the ass, what is written there in the configuration files of the applications that it serves.

And the database server can eat 2 gigabytes at the same time (namely RES, not VIRT memory)

Firstly, it's not so much if you only have 7.7G on the Physical Memory server. Secondly, where should the remaining memory go for some other applications? Thirdly, what's with the swap partition?

At startup, it consumes 1.4 gigabytes (RES) of memory, if you perform a soft reboot (graceful), then after the "rebirth" of processes, new ones begin to consume 3 gigabytes (RES) of memory.

Here it is possible that the operating system does not immediately reset the data pages, but thinks that they can still come in handy - a variant of a clock with two hands. In order not to go to the disk later for data in the event of a page out of memory error, the operating system does not immediately flush recent data to disk.

Why does the same software behave differently?

I myself am breaking my head over this question: two identical twin brother servers, the same distribution kit, but, bitch, they become different. Or the case - you put the driver for the video card on the machine, where there are two disks with identical operating systems (one in reserve). On one it becomes like clockwork, on the other it climbs like cancer, then in the end it ruins the entire system.