Why does the web server slow down?

Question

linux

Good afternoon. There is a nextcloud server on debian 11, with a postgresql database.
After switching to https, wild brakes began (in a couple of days) ... the site throws out a 502 \ 504 error
of Clients - 30 people, files in total 100GB.
300 "set-top boxes" on Linux, which in N hours request different folders for synchronization, using rclone, via webdav . A
common calendar is also configured, tied to iOS\Android smart phones of these 30 users.
The problem started, if I understood correctly, after switching to https... (which was required to enable sync calendars)

Used nginx, everything was fine. Now, once every N days\hours, php-fpm goes on a "spree", loading the CPU wildly (50% of the 8-core Intel), and puts the web server ... in the NGINX logs - connect time, which is logical ... in php-fpm logs - silence

I decided that the problem was with NGINX, quickly transferred everything to Apache.... The same crap, a bunch of Apache processes are created, there is a big load. But the Apache at least somehow works .. lets it creak. Moreover, after the first login - everything flies further ...

postgresql 13
php - 8 version of
apache2 and nginx - the last available, for debian11 in standard repos ...
The processor is free, the RAM is dofiga, everything is ok with disks, the network is not
loaded brute force (built-in from nextcloud) firewall closed everything except ports 80 and 443 (and ssh to my IP)

Help someone with what you can, I have already broken my whole head ...

Apache SSL config

spoiler

ServerName domain.ru
ServerAlias www.domain.ru
# ServerAdmin [email protected]
DocumentRoot /var/www/html/nextcloud

ErrorLog ${APACHE_LOG_DIR}/domain.ru.log
CustomLog ${APACHE_LOG_DIR}/domain.ru.log combined


Options FollowSymlinks
AllowOverride All
Require all granted


SSLCertificateFile /etc/letsencrypt/live/domain.ru-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.ru-0001/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

NGINX config for the site

spoiler

server {
listen 80;
# listen [::]:80;
server_name domain.ru;

# Add headers to serve security related headers
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;

#I found this header is needed on Ubuntu, but not on Arch Linux.
add_header X-Frame-Options "SAMEORIGIN";

# Path to the root of your installation
root /var/www/html/nextcloud/;

access_log /var/log/nginx/nextcloud.access;
error_log /var/log/nginx/nextcloud.error;

location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}

# The following 2 rules are only needed for the user_webfinger app.
# Uncomment it if you're planning to use this app.
#rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
#rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
# last;

location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
return 301 $scheme://$host/remote.php/dav;
}

location ~ /.well-known/acme-challenge {
allow all;
}

# set max upload size
client_max_body_size 512M;
fastcgi_buffers 64 4K;

# Disable gzip to avoid the removal of the ETag header
gzip off;

# Uncomment if your server is build with the ngx_pagespeed module
# This module is currently not supported.
#pagespeed off;

error_page 403 /core/templates/403.php;
error_page 404 /core/templates/404.php;

location / {
rewrite ^ /index.php;
}

location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
deny all;
}
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
deny all;
}

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {
include fastcgi_params;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
try_files $fastcgi_script_name =404;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
#Avoid sending the security headers twice
fastcgi_param modHeadersAvailable true;
fastcgi_param front_controller_active true;
fastcgi_pass unix:/var/run/php/php8.0-fpm.sock;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
}

location ~ ^/(?:updater|ocs-provider)(?:$|/) {
try_files $uri/ =404;
index index.php;
}

# Adding the cache control header for js and css files
# Make sure it is BELOW the PHP block
location ~* \.(?:css|js)$ {
try_files $uri /index.php$uri$is_args$args;
add_header Cache-Control "public, max-age=7200";
# Add headers to serve security related headers (It is intended to
# have those duplicated to the ones above)
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;
# Optional: Don't log access to assets
access_log off;
}

location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {
try_files $uri /index.php$uri$is_args$args;
# Optional: Don't log access to other assets
access_log off;
}
}
server {
listen 443 ssl http2;
# listen [::]:443 ssl http2;
ssl_certificate /etc/letsencrypt/live/domain.ru/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/domain.ru/privkey.pem;
# Add headers to serve security related headers
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;

#I found this header is needed on Ubuntu, but not on Arch Linux.
add_header X-Frame-Options "SAMEORIGIN";

# Path to the root of your installation
root /var/www/html/nextcloud/;

access_log /var/log/nginx/nextcloud.access;
error_log /var/log/nginx/nextcloud.error;

location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}

# The following 2 rules are only needed for the user_webfinger app.
# Uncomment it if you're planning to use this app.
#rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
#rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
# last;

location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
return 301 $scheme://$host/remote.php/dav;
}

location ~ /.well-known/acme-challenge {
allow all;
}

# set max upload size
client_max_body_size 512M;
fastcgi_buffers 64 4K;

# Disable gzip to avoid the removal of the ETag header
gzip off;

# Uncomment if your server is build with the ngx_pagespeed module
# This module is currently not supported.
#pagespeed off;

error_page 403 /core/templates/403.php;
error_page 404 /core/templates/404.php;

location / {
rewrite ^ /index.php;
}

location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
deny all;
}
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
deny all;
}

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {
include fastcgi_params;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
try_files $fastcgi_script_name =404;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
#Avoid sending the security headers twice
fastcgi_param modHeadersAvailable true;
fastcgi_param front_controller_active true;
fastcgi_pass unix:/var/run/php/php8.0-fpm.sock;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
}

location ~ ^/(?:updater|ocs-provider)(?:$|/) {
try_files $uri/ =404;
index index.php;
}

# Adding the cache control header for js and css files
# Make sure it is BELOW the PHP block
location ~* \.(?:css|js)$ {
try_files $uri /index.php$uri$is_args$args;
add_header Cache-Control "public, max-age=7200";
# Add headers to serve security related headers (It is intended to
# have those duplicated to the ones above)
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;
# Optional: Don't log access to assets
access_log off;
}

location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {
try_files $uri /index.php$uri$is_args$args;
# Optional: Don't log access to other assets
access_log off;
}
}

UPD in comment

Answer 1

In general. after increasing the network speed by 1000mb and commenting with RewriteEngine in the apache config, the errors disappeared ... php-fpm was also removed, everything works on the mod_php module
.
commented out the lines)

spoiler

ServerName domen.ru
ServerAlias www.domen.ru
# ServerAdmin [email protected]
DocumentRoot /var/www/html/nextcloud
ErrorLog ${APACHE_LOG_DIR}/domen.ru.log
CustomLog ${APACHE_LOG_DIR}/domen.ru.log combined
Options FollowSymlinks
AllowOverride All
Require all granted
## вот отсюда закоментил временно
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =domen.ru [OR]
#RewriteCond %{SERVER_NAME} =www.domen.ru
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

apache ssl

spoiler

ServerName domen.ru
ServerAlias www.domen.ru
# ServerAdmin [email protected]
DocumentRoot /var/www/html/nextcloud
ErrorLog ${APACHE_LOG_DIR}/domen.ru.log
CustomLog ${APACHE_LOG_DIR}/domen.ru.log combined
Options FollowSymlinks
AllowOverride All
Require all granted
SSLCertificateFile /etc/letsencrypt/live/domen.ru-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domen.ru-0001/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

The point is that I need http and https to work, because there are many clients that are on http, they have not yet been reconfigured.
If someone knows how to do it better / more correctly, I'm waiting for advice

Answer 2

Look at the server logs, Apache logs, run htop and see if anything else can load the server. It can be anything from a tuning curve to DDoS.

Answer 3

So Apache also swears to you

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

, and Nginx swears

[mpm_prefork:error] [pid 70323] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

on the same thing, but you don’t see point-blank?
The database also swears

HINT: See server log for query details.\nCONTEXT: while rechecking updated tuple (4,118) in relation "oc_file_locks"

, one query updates the records (and thus locked the tables), other queries wait for the lock to end and cannot wait.
By the way, $1, $2, etc. are visible in the logs. Did you replace it, or is this how the database sees the request? Or is it a mistake in the code, and the required values ​​​​are not substituted into the request at all?
Also, why "lock" instead of the kosher SQL `lock`?

Answer 4

What HTTP requests slow down?
What file operations do these requests cause?
What happens to the database at this time?
How many simultaneous requests from clients during such loads?
I would try to get graphs of various parameters over time, from which you can build on to solve the problem.
You can install some munin-node or other monitoring systems.