How to synchronize two file servers?

Question

linux

Please tell me how to solve the problem. Two servers. They are in different parts of the city. They work under debian. They act as a file server for the office. The data must be kept up to date. The volume of data is more than 6 TB. The whole problem is that the data is completely different. There are folders in which there is a very deep tree of subfolders and in each folder there are many small files. The total number of files in such folders is more than 2 million. And there are folders with images of working virtual machines with volumes of several tens of gigabytes per machine. Also a bunch of DVD images and other garbage. Tried both Syncthing and sync from torrent. Everyone falls into an incomprehensible stupor. If I let synchronize the storage root as a point, then they fall out with errors after several hours of work. As I understand it because of the number of files or volume. Maybe there is a solution? It is desirable, of course, that it be via http so as not to raise tunnels. But if there is a solution, then I will also try it. Thank you.

Answer 1

Try RSYNC. He seems to have the most advanced synchronization algorithm.

Answer 2

1) configure ssh authorization by keys
2) rsync or unison - both can transfer files via ssh(sftp)
3) run in cron

Answer 3

rsync option, other crutches too, but I would suggest Ceph

Answer 4

csync