Why did the web server take a long time to respond to requests?

Question

MySQL

Good day.
There is a web server on a virtual machine.
Recently, he began to slow down.
ubuntu system 12.apache2 + mysql. Proftpd is also spinning, which I turned off for verification and it didn’t help (it means it’s not about it)
The news site and forum are spinning on the server. Attendance is very low. Until last weekend, the site worked great. There were no complaints or complaints.
On Monday, it turned out that the site responds to requests for a very long time - more than 10 seconds.
There is nothing special in the logs (most likely I just don’t understand which of these logs can be used).
htop shows that the percentage and memory are loaded no more than 10% (even rarely reaches 5%).
ifstat also does not show any bad (not normal) network activity.
I'm just starting to delve into the structure of web servers and how to monitor (explore) them, so please explain in more detail:
How and where to enable such logging that will give me the necessary data?
Where should you look first?

Answer 1

1. Enable slow query logging in MySQL
2. View the Network tab in your browser

Answer 2

openvz?

Answer 3

iotop - disks can "fall in"

Answer 4

iotop -oka

Answer 5

iotop-oka

within 3-5 minutes (I opened the pages of the site abundantly) I accumulated the following:

spoiler

Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
9125 be/4 mysql 12.00 K 248.00 K 0.00 % 1.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
8653 be/4 mysql 0.00 K 7724.00 K 0.00 % 0.37 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
236 be/3 root 0.00 K 2096.00 K 0.00 % 0.27 % [jbd2/dm-0-8]
8646 be/4 mysql 0.00 K 0.00 K 0.00 % 0.22 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
8644 be/4 mysql 0.00 K 0.00 K 0.00 % 0.21 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
8647 be/4 mysql 0.00 K 0.00 K 0.00 % 0.20 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
8645 be/4 mysql 0.00 K 0.00 K 0.00 % 0.18 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
9066 be/4 mysql 8.00 K 256.00 K 0.00 % 0.12 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
9583 be/4 www-data 8.00 K 136.00 K 0.00 % 0.10 % apache2 -k start
8671 be/4 mysql 16.00 K 248.00 K 0.00 % 0.04 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
8639 be/4 mysql 0.00 K 0.00 K 0.00 % 0.01 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
9600 be/4 www-data 8.00 K 76.00 K 0.00 % 0.01 % apache2 -k start
9568 be/4 www-data 4.00 K 248.00 K 0.00 % 0.01 % apache2 -k start
9571 be/4 www-data 4.00 K 640.00 K 0.00 % 0.00 % apache2 -k start
9422 be/4 www-data 4.00 K 20.00 K 0.00 % 0.00 % apache2 -k start
1728 be/4 root 0.00 K 0.00 K 0.00 % 0.00 % [flush-252:0]
8654 be/4 mysql 0.00 K 4.00 K 0.00 % 0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
9276 be/4 www-data 0.00 K 72.00 K 0.00 % 0.00 % apache2 -k start
9599 be/4 www-data 0.00 K 484.00 K 0.00 % 0.00 % apache2 -k start
1787 be/0 root 0.00 K 8.00 K 0.00 % 0.00 % atop -a -w /var/log/atop.log 600
9566 be/4 www-data 0.00 K 12.00 K 0.00 % 0.00 % apache2 -k start
9570 be/4 www-data 0.00 K 252.00 K 0.00 % 0.00 % apache2 -k start
9601 be/4 www-data 0.00 K 32.00 K 0.00 % 0.00 % apache2 -k start
9625 be/4 www-data 0.00 K 32.00 K 0.00 % 0.00 % apache2 -k start
Where to look then, brothers? what logs to see? :( FORGOT TO SPECIFY
IMPORTANT:
Apache has 2 sites-enabled
Forum - flies without millisecond delays.
News (a set of pages with info and documents) - opens any of the pages for 10-11 seconds.
It is Apache that renders pages at such a speed.