Abnormal load on the server - how to find the cause?

I

Iskander Giniyatullin2011-06-16 14:08:35

linux

Iskander Giniyatullin, 2011-06-16 14:08:35

In general, the alignment: there is a server on debian, Intel Pentium D 2.8 GHz, RAM 4G, 3 x 160 Gb
It has a standard set: mysql + php + nginx + apache + sphinx + exim + dovecot

It has 3 sites running on it, with a total attendance of about 20k hits per day.

Usually (before) the load on the server (la) did not exceed 2, when heavy things were done on the cron - about 4.

Today, anomalies began: after a reboot, the server works fine for some time (about 20 minutes), and then problems begin - LA grows like on yeast: 10… 20… 30… reached

110 do
/etc/init.r/apache2 stop
/etc/init.r/mysql stop
/etc/init.r/exim4 stop
/etc/init.r/nginx stop
/etc/init.r/cron stop

Then LA drops from 20+ to 4-5, but not lower

. How can this be? What eats up server resources when, it would seem, everything is turned off? How to find out?

Reply

Answer the question

In order to leave comments, you need to log in

7 answer(s)

H

homm, 2011-06-16
@rednaxi

rednaxi, in the provided screenshot, the searchd process is clearly visible, which is eating 750 megabytes of memory and 2.5% of the CPU (and god knows how many hard disk accesses). Is this the Sphinx? And who puts their sessions in /var/www/beboss/data/mod-tmp/ and how many files are there now?

E

Elkaz, 2011-06-16
@Elkaz

htop

V

vbuoc, 2011-06-16
@vbuoc

Damn, I didn’t see the link to the screen ...
I'm sorry, the end of the slave. day...

V

vbuoc, 2011-06-16
@vbuoc

I think the problem in mysql is 85% sure you need to dig there!

G

GANGTA, 2011-06-16
@GANGTA

Anonymous visitors :)

I

Iskander Giniyatullin, 2011-06-16
@rednaxi

Additional information: today in the middle of the night the server suddenly hung, after starting to swear about Too many connections
After that, it had to be rebooted, which led to file system damage - I had to do fsck -y to start it again
After that, it turned out that some tables were marked as crashed - I had to do myisamchk and restore them
. After that, the server sort of got up and everything worked fine, but not for long - after about 20-30 minutes of work, the symptoms that I described appeared.
I found a similar problem on Habré: Debian server dies when MySQL builds an index. I also have one table on the site for 3.5 million records, with indexes for 500 megabytes, which is quite actively recorded (statistics), and besides, it also crashed today. I tried to cut off a record in this table - the situation became better, after 40 minutes of the LA server's operation "only" 5-6.
However, before the described events (reboot) everything worked fine with the old settings.

V

vbuoc, 2011-06-16
@vbuoc

Looking forward to what the author has to say...