Disk replication via DRBD over a slow network, or another solution?

Question

linux

Good afternoon.

There are two servers connected by a slow network (100Mb/s or worse, over the real internet).
One of the servers is the main one, from the second I want to make something like a "warm" replacement.

On the server, in the end, they are assembled, linked into a tricky system, and different virtual machines are spinning, on which users work around the clock, but at night - very little, and sometimes not at all.

Virtual machines are cut into logical volumes from LVM, LVM is laid on the original partition with a size of, say, 2 TB.
I would like to make sure that this partition is constantly copied to the backup server "as is", with all the data and LVM metadata inside.

The logical solution seemed to be DRBD. A resource was created, on the main server it was declared Primary, on the backup - Secondary. In the event of a failure of the main server, it is supposed to simply turn on the backup server, declare the Primary section on it and start all the virtual machines.

Technically everything worked, but the problem turned out to be in the network - it is slow. In the test example 100Mbit/s, in reality it will be 30-50. All changes that are written to the Primary section are trying to get dragged into the Secondary online. As a result, the write IO speed on the Primary drops to 11MB/s, and the latency is also too high. For a moment, the physical disk itself is an SSD, which will take 500MB / s, and here - 11.
And on one of the virtual machines - a 1C base that needs high write speed. With 11 MB/s it just won't move.

I would be satisfied with the option in which Primary would receive data from users at the speed of the disk subsystem, and they would be transferred to Secondary as far as possible. Let's say if the stars converge once a day or two, and the synchronization catches up with the current state of the disk, it will suit me. Volatile data is copied in a different way, the team of virtual machines and operating systems on them are valuable. To ensure the consistency of Secondary, DRBD can create snapshots between moments of full synchronization (I have not tested it yet, and this is not in the config).

I can, for example, turn off Secondary, and turn it on only at night. Then, when Secondary is disabled, the write speed on Primary is just excellent, and then Secondary is enabled and takes the difference. But then there will be a low recording speed at night, which will lead to users' discomfort.

It seems that, since 8.4, DRBD is able to use the delayed synchronization mode AHEAD / BEHIND, which allows Primary to go ahead, due to the temporal inconsistency of Secondary. But for the life of me, I can't set it up. All the same, recording on Primary is carried out at network speed, and the devices are synchronous - there is no break in AHEAD / BEHIND.

Help me please. I broke my whole head. What am I doing wrong?

Test environment config:

resource data {
    device /dev/drbd1;

    syncer {
        rate 80M;
    }

    startup {
        wfc-timeout 10;
        degr-wfc-timeout 10;
    }

    on XVMS0 {  // тут Primary
        address 192.168.0.248:7789;
        disk /dev/sdf;
        meta-disk /dev/vgaux/drbd_meta_sdf;
    }

    on XVMS1 {
        address 192.168.0.249:7789;
        disk /dev/vgmain/drbd_test;
        meta-disk /dev/vgaux/drbd_meta_test;
    }

    disk {
        al-extents 25600;
    }

    net {
        protocol A;
        on-congestion pull-ahead;
        congestion-fill 10G;
        congestion-extents 100;  // пробовал и большие значения, ближе к al-extents.
        data-integrity-alg sha1;
    }
}

Maybe there are other solutions, not DRBD?

I have my script to differentially sync block devices over the network. It would be perfect if synchronization was needed once a month. The problem is that it leads to the subtraction of both devices, which is unreasonably long for a 2 TB HDD, almost 10 hours. I want exactly that DRBD chip that marks the recorded areas for transferring the difference, and does not reveal it by parallel reading of two devices.

Thanks in advance.

Answer 1

Everything is sad, in short, and bad.
DRBD will work in my desired "fully asynchronous" mode only through the DRBD-proxy. Which is a commercial product from LINBIT, and it costs so much that they don't even put out a price tag, but prefer to put up commercial ones by mail.
According to scattered information - a few thousand dollars a year. For my case, it is unreasonably expensive.
I came to the conclusion that I will increase the size of the active area, and I will connect a secondary node at night, when almost no one is working. Through a gigabit link, because spreading nodes over the Internet will have to be abandoned due to the fact that synchronization over the night, it is highly likely, will not have time to complete.
I did not find other solutions that allow block devices (not files) to be synchronized differentially.

Answer 2

I won’t help with online file system replication, but to significantly (by orders of magnitude) speed up the process of backing up and getting diffs - using btrfs and its snapshots Take
regular snapshots, at least per minute (but it’s better to integrate their creation somehow into the application logic, for example, when it doesn’t happen records, since the moment of creating a snapshot does not take time, this will not significantly affect the operation of the program, but the snapshot itself will contain consistent data), then compare the earliest unsent with the latest on primary:

btrfs send --no-data -p /snapshots/parent /snapshots/child

get strem, which you send to the backup server and deploy:
btrfs receive /backup/snapshots
https://serverfault.com/questions/399894/does-btrf...

Answer 3

Virtual machines kvm?
Make replication like in proxmox for example

Answer 4

1) I am very surprised by what the author is doing. It is categorically impossible to backup databases through image replicas. Or at this time the database needs to be stopped. There is a risk that everything that the author has accumulated will be useless rubbish. after restoring the database will not rise. There will be a lot of corrupted block. And inconsistent data.
Therefore, the question is - the author did you even try to restore the entire complex from such an inconsistent backup?
2) If the 1C database is on MySQL or on PG, then you need to use the box dump utilities my*, pg* dump.

Answer 5

Hello!
From simple to complex, try the following:
1. Reset bios settings to default.
Go into BIOS (F2 or DEL), click reset, save and exit.
Press F12 or F11 to bring up the boot selection menu. Check if the flash drive is visible.
2. In the BIOS, change the mode from EUFI or Legacy to =>> Dual Boot.
3. Enable USB recognition in BIOS
Exactly where this feature is located in your BIOS is not known, but it is somewhere.
We want to overcome this problem!