How to implement a scalable Websocket server architecture?

Question

Redis

There is a websocket server and redis db in which the id of all connected clients are saved. There can be many Websocket servers. Clients (browsers) use a load balancer to connect to different websocket servers. Clients (browsers) can request the id of all clients from the websocket server, the websocket server requests this data from redis db and gives it to the client (browser).
The question is what to do if some instance of the websocket server fails (for example, the power is turned off)? Established connections were broken, but the client ids of the failed websocket server are still stored in the redis db, and clients connected to other websocket servers now receive incorrect information. How to give clients up-to-date data (client id) when emergencies occur that cannot be processed programmatically on the machine on which the emergency occurred?

Answer 1

We monitor emergencies by monitoring. As soon as the server falls off, we process the corresponding event (in your case, as I understand it, this will be a data correction). The speed of response depends on how quickly monitoring learns about a failure. In any case, there is some time when clients may receive outdated information.
Failures are different. Process crash, OS crash, equipment failure. This requires a certain threat analysis for your system. Monitoring should work out all critical situations.
Another possible solution may be to raise a new server from the hot standby, which will take over the work with the data of the "extinguished" server. But these are minutes of no server response...

Answer 2

And what prevents a separate daemon from doing that keeps connections with all websocket servers, and as soon as the connection with the server is lost, it cleans up the database and tells the balancer not to use this server temporarily.
Ideally, the balancer itself should do this, constantly be in touch with the servers and know which of them are not working and in general how much each of them is loaded at the moment.