Distributed cluster on Proxmox. What do you advise?

Question

Proxmox

Good afternoon!
There is a task to make a distributed system. The virtualization environment was chosen by Proxmox. Geographically, not only physically separated by different cities of the server.
What you want:
1. That each server would pull its part of the systems, conditionally half.
2. If there is a fall or stop of one part, then the second one takes over. Not necessarily instantly, I think at the first stage it will be possible to restore from a backup of machines.
3. Well, the final stage is offline work and minimal admin intervention.
Questions:
1. How much Cheph do you need?
2. Set up a Proxmox cluster and what are the nuances?
3. Traffic between nodes - tips for organizing a channel?
4. Advice on organizing a shared storage or will it be enough for each node of its own?
5. In general, what are the pitfalls for a distributed system?

Answer 1

Questions:
1. How much Cheph do you need?
2. Set up a Proxmox cluster and what are the nuances?
3. Traffic between nodes - tips for organizing a channel?
4. Advice on organizing a shared storage or will it be enough for each node of its own?
5. In general, what are the pitfalls for a distributed system?

First, I will answer the questions, and then I will supplement a little on your situation.
Ceph is needed in your position and will allow you to avoid the cost of expensive storage systems. The question of what is better than CEPH or storage is more rhetorical. There are failures here and there, and mainly from ignorance of the components. I can say one thing, cpeh is ready for production!
A cluster on proxmox has only probably one and not immediately obvious nuance, multicast support is needed.
Traffic between nodes must be within the same DC. As in general, the ceph storage.
Ceph as the storage of the VMs themselves must have 10gb or infinity band , I do not offer faber channel, since it is outdated. (from my point of view).
Prob shared storage, the question is a bit confusing.
If you are making a cluster, then in any case you will already have data storage somewhere outside.
As an option, 3 servers that will hold the VM and 1 storage / CEPH server.
It is clear that here the point of failure will be 1xd and it must be increased to at least 3x. Further, the point of failure will be network equipment, etc. Building depends on how much money you can invest and what risks you are willing to take.
Now there will be an answer to the fifth question and my vision.
If you have some kind of application, then it is usually made fault-tolerant and they try to spread it on different DCs, etc. With virtual machines, this number does not work, alas, they all live in the aisles of their DC. Yes, of course, you can conditionally drag it to another. This is roughly similar to the work of a VM in amazone, digitalocean and migration from region to region. It is quite possible to do this, but the configuration itself in the aisles of the DC does not go anywhere.
I will ask questions.

Answer 2

1) how do you want to send data?
2) depending on what kind of application and database you have, I would not advise raising geo on proxmox.
3) Do you have the opportunity to buy an optical link between DCs? it's just very expensive
4) depends on applications
5) full of pitfalls.