A large number of open files in the system?

Question

linux

Some strange problem. For some reason, in the system, over time, the number of open files rests against all possible limits. First in ulimit - increased:
[email protected]:~# ulimit -a |grep "open files"
open files (-n) 900000
[email protected]:~#
Then in fs.file-max, increased:
[email protected]:~# sysctl -a|grep fs.file-max
fs.file-max = 99999999
[email protected]:~#
Not much of a problem right now, but with file-max set to 200000, it took about half a week after server reboot to get there numbers.
As a result, now I have a number of open files, according to the system, about 223000:
fs.file-nr = 222528 0 99999999
At the same time, there are only 1700 entries in lsof.
Axis Ubuntu 11.04, regular bucket 2.6.38-8 x86_64. Apache is running on the server as a backend for nginx (which is located on another server), there is also an icecast radio server (clients do not cling to it), an audio decoder and an nfs server for serving static content with nginx from another server.
The real question is - what is it? What can be done about it? How to find out what these 200.000 files are, except through lsof? You can, of course, randomly change the cores and the axis, but I want to know the reasons first.

Answer 1

NFS update. True, it is not clear where - only experimental versions are ahead. But, nevertheless, the bug is known. We encountered it when ispmanager CE was deployed.

Answer 2

netstat? )

Answer 3

Let's start with lsof | wc -l
and in general man lsof
In my practice there were similar cases, all of them ended with an open mail relay, although there was still trouble Apache opened over9000 files from something.

Answer 4

Read this thread here.

Answer 5

The first thing I would try is not to distribute static over NFS, there were mentions in the nginx mailing list that this could have unexpected unpleasant consequences. Put nginx on this server, and on the frontend just make a proxy_pass to it, and see what happens.