How can you create workers and balance the Telegram bot?

Question

PHP

There is a moderator bot, which consists of a large number of groups and moderates them for the presence of forbidden words, everything worked well before, the number of requests did not exceed 30, but now it is about 3000 requests per second and one server on which everything used to be (nginx + php stack -fpm + mysql + redis, as well as the admin panel) simply can’t cope, so the following questions appeared:
1) Mysql heavily loads the system. Due to frequent operations with the database (the user changed his name, left some message, changed the message, etc., it is necessary to store, so almost every webhook request is at least one request to the database), a php script can be generated, which is webhook, which causes telegrams to send even more webhooks because they did not receive a response from the previous ones, because they did not receive a response from the base. The data needs to be updated in real time, so the question arose of how best to do this? Transfer database to another server? Or will it not help the situation and make things worse?
2) One access point is also a bottleneck in the system. All bots are tied to it (for convenience, group admins can create personal bots and add them to confs), there was an idea to send all requests to rabbitmq, which would be parsed by workers, but as I understand it, the script should be launched by a daemon and request a request from the queue itself? Is it possible to implement this so that on the side where the webhook goes, its verification takes place, if everything is OK and this bot is assigned to some administrator - send the webhook to one of the workers?

Answer 1

The neck in the logic and in the database most likely.
First, you need to make a balancer (on nginx).
It already balances traffic to php-fpm on other servers.
php-fpm itself already does the logic. Redis is unlikely to be hard loaded. But with the base there can be troubles. Still, 3k rps (most likely you have several requests to the database for each hook). The easiest way is to make a master-slave configuration.
In total we have:
1) server for the balancer
2) servers for php-fpm (at least a couple)
3) redis server (at least one) 4
) master mysql
server 5) slave mysql server
case, network delay will greatly worsen the situation)
This can be done in any cloud.
Well, of course, each type of server has its own hardware config.
For example, muscles need fast disks and preferably more RAM. For radish, you also need fast disks (if you throw off the base on the disk). There are no special requirements for the balancer, but it is advisable not to offend the server in terms of RAM and CPU. For backends (php-fpm), you need to focus on the processor + RAM to your liking (depending on the logic). In general, the configs will need to be tested and twisted. Therefore, the cloud and I advise.