Web-failover in another data center.?

Question

Nginx

At the moment, for load balancing on web nodes, we use the following scheme: CARP + 2 x HAProxy + 10 x Nginx. The big disadvantage of this scheme is that CARP requires both balancers to be in the same network, respectively, on the same tech. site... In the recent past, the entire site fell down along with two balancers, which made the nodes located outside it inaccessible.
Is there anyone who wants to share their experience in organizing a failover on another site and automatically switching to it?

Answer 1

MPLS over GRE, for example.
Well, or, if it’s simpler, then OpenVPN can stretch ethernet through a TAP device.

Answer 2

You can play with bgp.

Answer 3

One option: Failover using DNS.
DNS servers check the availability of the site and, in case of a fall, switch the record to another site.
Cons - not fast even with small TTLs - dns records can hang in caches (browsers, local resolvers).
Experience: 90% of users switch within 2*TTL, 99% within an hour.