How to cure a server that thinks it's all out of disk space since df -h shows 100% used space on / but df -ih shows 27%?

Question

linux

Ubuntu 16.04 LTS (GNU/Linux 4.4.0-53-generic x86_64)
lsof +L1 shows nothing.
Reboot doesn't help.
sudo touch /forcefsck and reboot don't help.
df / -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       144G  138G     0 100% /

df/-hi

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/vda1        9.2M  2.5M  6.8M   27% /

sudo du -hcs /

138G	/
138G	total

In fact, it seems to me that exactly 27% should be occupied.
[email protected]:~# du -sh /*
paste.ofcode.org/6PLHCDjKBLt6WB7xBrxbcb
ncdu /
paste.ofcode.org/sfeWyMDWSpptu6YMQGsf8Q

Answer 1

Thank you all for your help! However, I got tired of sorting it out and (thankfully there was such an opportunity) decided to copy everything more or less important, so that later I could demolish everything and reinstall it.

Answer 2

Because the anodes are index descriptors, and the place for them is reserved immediately. Apparently you see how much space the system used for them and how much is left. Google before asking....

Answer 3

Show du -sh /var/*
There is a feeling that some remote file is holding the place. Try to stop and start (namely stop/start) nginx, apache and mysql.

Answer 4

A fairly classic case, the process holds a deleted file, the place is not cleared, but the file is no longer there.
It is searched like this: lsof -nP | grep '(deleted)'
It is treated by killing the process, or by cleaning the proc file.
There is a discussion on the stack.

Answer 5

First, do
sudo tune2fs -m0.1 /dev/vda1
And get more free space
Then figure it out.

Answer 6

Have you run a file system check on this partition?

Answer 7

Before calculating the occupied space, bind the root to an empty directory, because. other filesystems are mounted on top of the root, which can hide files in certain directories.
First
And then you analyze /any/non-system/folder

Answer 8

There is such a tool ncdu
https://dev.yorhel.nl/ncdu
This is an analogue of baobab for working without x.

Answer 9

Found a solution to this problem with digitalocean.
there is a hidden folder of pm2 logs that needs to be deleted
rm -rf ~/.pm2/logs/*