How to set up a WordPress server for high load and DDOS?

Question

Nginx

There is a dedicated server with Plesk Panel:
Intel Core i7-2600 Quad Core
32 GB DDR3 RAM
2x 3 TB SATA III HDD
The server has PHP 7.3 running on Apache+FPM+Nginx. You can enable pure NGINX, but you really don’t want to constantly fiddle with manual configuration for all WordPress (there are many WordPress sites installed on the server and you need .htaccess and all plugins dependent on them to work). WordPress has W3TotalCache installed and configured.
The essence of the problem: under normal load, everything flies, even with high attendance. But websites periodically get DDOS attacks from competitors, as I understand it, from manually controlled bot networks - as soon as I find a way to block the current attack option, they change it to another one.
The server is configured with fail2ban rules in which I periodically change to adapt to the changing attack.
Now the sites are on an old server with approximately the same hardware, but older software (Apache FCGI, PHP5, without Nginx):
Blocking common attempts to "swoon" from the same ip is easy with fail2ban, but the attacks turn into more and more sophisticated :
1) Starts hammering random requests to non-existent pages in WordPress, and since this does not give 404 errors (WordPress tries to find such articles and generates its own 404 page), then no cache helps here - each request generates a new caching. Sample requests (tens per second):
domain.com/DSADAJKFJASFJASFLKJ
2) IPs change constantly, they practically stopped using repetitions (each new request from a new IP).
3) The referrers were fake (all different), sometimes they use an empty one, after they began to be banned by my settings.
4) Sometimes they just hammer some page with a request (for example, sorting goods by rating).
5) Pound /wp-admin/admin-ajax.php with POST / GET requests - this greatly hangs the server. I can't completely block the file, because the frontend uses requests for it from real visitors. It turns out to block through .htaccess if they use an empty referrer. If not, it is not clear how it differs from normal users.
6) The domain is connected to Cloudflare, but as I understand it, with such types of attacks (where all ips are different and it is difficult to determine that the actions are dubious), it will not help much.
Actually the question is - as I understand it, setting up a ban for such attacks is automatically very difficult (especially as soon as they see that a ban is in progress, they immediately change requests, referrers, etc.), so another request arises - how to properly configure and set up a new server (where will I transfer the sites from the old one where the attacks are now), so that such attacks do not hang up the server, and it even works fine under them? As I understand PHP7 and Apache FPM and NGINX should help with this? What else would you recommend to install and configure? (Accelerators for PHP, databases?) Will switching to pure NGINX without Apache really help?
What Apache / Nginx settings should be specified (connection limits, etc.) - so that the server does not fall and everything works for everyone? (there were no 503/502 errors).
The sites themselves that are attacked after creation are practically not updated with new information (so full deep caching is possible), but have some dynamic features (like post likes) that should still work.
Tell me what software bundles should be used to make it all fast despite the load? Will the specified server hardware be enough?

Answer 1

If I were you, I would completely switch to Nginx + php-fpm.
On nginx, you can limit the number of requests per second to the site using this module, you can also hard cache pages using Nginx , and you can also restrict access from unwanted ip addresses (for example, from Africa, China and other unnecessary countries).

Answer 2

if the site allows, set up more aggressive caching. Give everything as static, at least during attacks.
This can be done both with plugins (for example, the same Wp Super Cache), or with nginx settings.
Below are tips, as if you had already set everything up for static feedback.

1) Starts hammering random requests to non-existent pages in WordPress, and since this does not give 404 errors (WordPress tries to find such articles and generates its own 404 page), then no cache helps here - each request generates a new caching. Sample requests (tens per second):
domain.com/DSADAJKFJASFJASFLKJ

why do you cache such pages separately?
give one static 404 page for all non-existent pages
several options:
1 add nounce (and make it work correctly with cached pages)
2 stop using admin-ajax.php on the front. replace with REST API/custom endpoint.
Yes, this will not be according to WP standards and will require some reworking of the code / plugins, but it will be more productive, at least due to the operation of a slightly different set of hooks for the rest api ( rest api vs ajax ). Plus, requests to the standard admin-ajax are poorly cached
, and with a custom endpoint, in general, you can only load the necessary parts of the engine
at a minimum, leave only the countries you need, block the rest.
during the attack, unscrew the Basic Protection Level to the maximum mode (I don’t remember from which tariff plan this is available)

Answer 3

Look towards haproxy as a frontend - there is a very flexible ACL language that allows you to do all kinds of rate limits, reset inactive connections, etc., I'm sure that it will help you repel at least part of the attacks.

Answer 4

The essence of the problem: under normal load, everything flies, even with high attendance. But websites periodically get DDOS attacks from competitors, as I understand it, from manually controlled bot networks - as soon as I find a way to block the current attack option, they change it to another one.
The server is configured with fail2ban rules in which I periodically change to adapt to the changing attack.
Now the sites are on an old server with approximately the same hardware, but older software (Apache FCGI, PHP5, without Nginx):.....

No need to sculpt skis from straw. Get protected. Worse than your attacks hold back.