Why does a 502 error occur every day from 9-11 am?

Question

linux

Hello.
there is a site and worked for 6 months in a closed mode (http authorization)
a site on Bitrix, code on modules, its tables and ORM (it worked as a CRM and orders were placed there in closed mode) a
few days ago the site was launched for people
and now it gives back every day 502 from 9 am to 11 am.
VPS server 4 cores, 4 gigabytes, ssd
CentOS + bitrixVM php7.1
(OPcache) + Apache/2.4.6 + nginx/1.16.1 + mysql in the erroneous httpd logs, only this [error] server reached MaxClients setting, consider raising the MaxClients setting is very interesting why always at the same time and why this problem did not exist before?
and how to fix it, how to win?
now I'm doing httpd restart and it starts working, it works for 40 minutes and goes to bed again, after 11 am everything is fine

Answer 1

htop at the same time does not show heavy loading at all, and the hoster panel does not show

1) I would check the graph of the number of hits to the site.
2) Maybe the hoster is running a backup of the database or the like at this time?
3) Activate slow query log in MySQL - maybe we will see something interesting

Answer 2

Hello!
There are several options for investigation.
You can tie third-party monitoring, something like https://newrelic.com/
You can arrange load testing on an environment close to combat. Take some kind of Apache Jmeter, write a script of the user's work with the site through it and load the test site. Look at how many users it goes to. See what is happening with the database and other components of the site.
You can screw on a distributed tracing system and see where the bottlenecks are.