Why is memory clogged with buff/cache and oomkiller kills virtual machines?

Question

linux

The bottom line is this, there is a server with such a config 2xE5-2630v3 / 8x4Gb RDIMM DDR4 / LSI 9261-8i / 8x4Tb SAS (the disks are pulled out of the HP MSA 2040 shelf and branded by HP), the OS is Ubuntu Server 16.04, the array is divided into 2 partitions, 2Tb / and 20Tb /home for the archive, both ext4 partitions.
A raid 50 array of 8 disks was assembled, the controller does not show degraded disks, there is no BBU, but the controller cache is enabled. If you turn off all services and test dd, then in linear recording it gives out ~ 700mb / s, in reading 1.4gb / s.
In the north, flussonic is spinning with an archive for ~ 300 threads, a MariaDB cluster node (~ 3k rqps) with a replica to a backup server, a GlusterFS node (2 volumes, without a replica yet, in one there are two small virtual machines for migration to another server, in the second layers of cartography for GeoServer) and several other virtual machines in KVM.
Recently, on this server, OOM Killer began to kill the desired virtual machine, tk. it takes up the most memory, while free shows used only 16GB of memory, the rest is clogged with buff / cache. Since the necessary virtual machine intensively loads the disk - I thought it was disk caching, turned it off - anyway, after half an hour or an hour, buff / cache takes 10GB. So far, I have thrown sync && echo 1 > /proc/sys/vm/drop_caches into cron every 15 minutes, which resulted in 100% loading of the array during the execution of the task.
I tried to tune vm.swappiness=10, vm.vfs_cache_pressure=1000, but it doesn't help, the load on the array in iostat is kept at 30%, the memory still runs out just as quickly. I set vm.min_free_kbytes to 8GB, as a result buff/cache took 8GB and the server completely crashed.
In what direction can you dig further? Or here I rest against the performance of the disk array and all that remains is to migrate some of the services to other servers?

Answer 1

the rest is filled with buff/cache.

www.linuxatemyram.com