What ate the hard drive?

Question

linux

Hello.
The web server running debain ran out of space.
I deleted the overgrown access logs, old backups, freed 10GB, but they were immediately eaten.
The hard drive is not big, 200 GB.
The web project on the server occupies ~ 1GB
Databases to it ~ 2GB
Replication is configured, the server acts as a master. Binary logs occupy a stable 32GB
In addition to the above:
Searched for files over 100MB

find . -type f -size +100M -exec ls -l {} \; | sort -k5 -n -r| less

Now it's just databases, binary mysql logs and one very strange (!) file /proc/kcore which takes up 134217726 MB on a 200 GB disk - What O_o?

Searched for directories that occupy more than others.
du -kx | egrep -v "\./.+/" | sort -n
The /chroot directory at the root itself stands out, which takes up a lot of space. Inside is an almost complete copy of the entire root directory, although everything is dated from the beginning of last year.
A few questions:
Where could these 10 GB disappear in a moment?
Where did 150GB go?
What kind of strange file, which is supposedly ten times larger than a railway in volume?
What can be removed?
Particularly interested in the chroot directory, can it be removed and will it affect something?
UPD:
Removed binlogs.
ncdu output shows that only 8Gb

is occupied

Answer 1

/proc/kcore is just an interface to a physical RAM, its size is usually 128TB. In general, you don’t need to look for anything in / proc, because it's a pseudo file system.
What is there in the chroot directory is more visible to the one who installed the system.
And some logs could gobble up the place. Another option - some process writes something to a file, you deleted this file, but the process continued to write. After rebooting the system, the result should change.

Answer 2

sudo du -hs /*
will see which directory the "trash" is in. Further, respectively, sudo du -hs / directory_name, etc.
there is.

Answer 3

Oh, and Google gives out a lot of interesting things, for example, here or here .

Answer 4

First, ncdu / will really help.
If it doesn't help, then - https://debian.pro/1654