Strange web server behavior on nginx+php-fpm?

Question

linux

Colleagues, hello.
The situation is a mystery.
There is a web server on nginx and php-fpm. Debian.
# nginx -v
nginx version: nginx/1.2.1
# php5-fpm -v
PHP 5.4.4-14+deb7u3 (fpm-fcgi) (built: Jul 17 2013 17:57:18)
Copyright © 1997-2009 The PHP Group
Zend Engine v2.4.0, Copyright © 1998-2012 Zend Technologies
with XCache v2.0.0, Copyright © 2005-2012, by mOo
# uname -a
Linux 3.2.0-4-686-pae #1 SMP Debian 3.2.46- 1 i686 GNU/Linux
# cat /etc/debian_version
7.1
# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 20G 3.3G 16G 18% /
udev 10M 0 10M 0% /dev
tmpfs 61M 80K 61M 1% /run
/dev/xvda1 20G 3.3G 16G 18% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 121M 0 121M 0% /run/shm All
of a sudden sites stopped working.
Restarting the nginx and php-fpm daemons did nothing. I sent the server to reboot, the sites started working. But not for long, then they stopped opening again. At the same time, nginx continues to work.
Outcome. After each reboot, the sites work for about five minutes and then stop opening.
Where and how to dig? The nginx logs do not raise any suspicions. I understand that there is very little initial data (it’s not all the same to throw in a row), but, to be honest, I’m a little in a stupor.

Answer 1

Friends, thank you all for your help and advice!
So, the problem is solved, was clearly on the side of Amazon AWS.
Turning off and on the instance (instance on EBS) from the EC2 console helped. Restarting the instance from the same did not help.

Answer 2

Understand in order.
1) static pages open?
2) calling localhost via "wget ​​-S ..." doesn't work either?
3) page with <?php phpinfo(); ?> returns anything?
4) what do the “free” and “ps axwwll” commands say at the time of the problem?
5) first check the MySQL settings through mysqltuner,
6) check the loading through mytop when problems start.

Answer 3

Not enough information.
Look at the php-fpm logs,
turn on the nginx logs, what are the errors,
see what processes are in the D state in the “ps axfu” output, what is the load in general and by what processes,
see if you have a stack of queries running in mysql (assumed by default).
Offhand, perhaps this is a php-fpm pool overflow caused by congestion in scripts (maybe heavy database queries or even a database error).

Answer 4

Usually this happens when some sockets on a virtual machine are overflowing, and they are truncated on it, well, or a neighbor grabbed them if we say cgroups or openvz.

Answer 5

If there is not even a connection to nginx, then the checks are as follows:
1) firewall? iptables-save
2) system limits? dmesg | tail
3) connection from this server to locahhost also hangs? wget -S -O /dev/null 127.0.0.1/
4) service nginx restart doesn't help?
5) if you add a minimum server on port 81 to the nginx settings, does it not open either?
About exhausting network sockets and file descriptors:
serverfault.com/questions/350643/exhausting-linux-machine-tcp-socket-limit-70k
serverfault.com/questions/154663/how-to-find-the-number-of-open -ports-in-linux
Enclose command output with "code"
чтобы столбцы не разъезжались, иначе ничего не понять.

Answer 6

Upload the full nginx configs somewhere on pastebin.
It is possible that they have some small blunder.
For example, instead of a limit of 4 workers, a limit of 4 connections is set.
It's still interesting to look at dmesg | tail -30 and iptables-save.

Answer 7

So, friends, nginx has been replaced by lighttpd.
And the situation remains the same, the site does not open on port 80.
Change the port to 81 and everything opens. The firewall is clean, see above. In the Amazon security console, the group is configured - ports 80, 81 are open.
How so? What can block access if, as they found out on another web server, the same thing.

Answer 8

Well, it's definitely fastvps and hetzner. Same thing every 2 days.
Stupidly set every minute crowns, if (in my case), the load on the top\u003e 2, then
service php5-fpm restart

Answer 9

The server in hatzner also faced a similar problem. It was not possible to find the reason. Only server restart helped.
The problem appeared after updating accounts in ISP Manager. Those. after restarting apache and nginx.
After updating the software, the problem seems to have disappeared.
Z.Y. Axis CentOS 6.4