What and how is the easiest way to automate the creation of backups of KVM machines?

Question

Python

There is a KVM hypervisor with 20 virtual machines. Every day you need to make full backups of some machines. While you have to manually turn off the machines using Virsh, make a backup, and then turn it on again.
Is there any tool besides bash scripts that can help implement the following script below?
1. Initiate virtual machine shutdown.
2. Make a complete copy of the virtual machine (I have them in the qcow2 format in the form of a single file, very convenient, by the way, I copied the file and you're done).
3. Turn on the virtual machine. 6. Start creating a backup of the next virtual machine (points 1 to 5).
4. Shrink the copied virtual machine (more precisely, its virtual hdd) so that the size becomes real (I do it using qemu-image).
5. Compress tar bzip.

Answer 1

I agree with 3ton about LVM. But doing a live backup is extreme.
You can get a half-updated database, a half-living file system, and so on. Except that you will make a copy not only of the disk but also of the memory at the time of the pause of the virtual machine.
Without access to the inside of the virtual machine, it is impossible to guarantee that all data has been flushed to the disk.
My advice - reconsider the technology of backups.
I practice the following approach:
Before updating the system on the server, I make a full backup according to the following scheme.

1. Stopping a virtual machine (virtual machines respond to ACPI from the outside)
   
2. Snapshot of the LV attached to the virtual machine.
   
3. Virtual start.
   When using scripts (not manually), the downtime will turn out to be almost equal to restarting the virtual machine.
   
4. Updating (of the system) If the kernel is being updated, then I do the second LV snapshot in a similar way.
   
5. copying snapshot(s)
   
6. deleting snapshots.
   

Keep in mind that it is necessary to have a reserve of space for a snapshot for the duration of its life (copying), since the free space on LVM will quickly begin to melt as soon as the virtual machine begins to actively change the data on its disk.
That is, it makes no sense to make a daily backup copy of what has not changed (the system and programs).
And I make daily backups using scripts + bareos/bacula.
This approach generally reduces the size of backups.
Well, it's good to restore and verify it after creating a backup.
For a virtual machine, deployment on another physical server is usually enough (so that there are no conflicts) and checking the running of all important services and database availability (test scripts).
Then you can optimize. Do not back up swap virtual machines. It can be removed when copying a snapshot. And it is better to give out swap to separate LV. Then you don't need to snapshot it.
In virtual machines, by analogy with a separate swap, I also make separate disks for DB, logs, and so on. All this allows you to shrink the size of the system backup to 3-5 gigabytes in uncompressed form.

Answer 2

Look towards bdsync and bdsync-manager You
may have to finish something, but where without it

Answer 3

At one time (5-6 years ago) I also asked this question, it was decided to do it on LVM. Now everything is simply solved by an LVM clone and I don’t even turn off the virtual machine. I have another question for you, but how many cores do you have? If I'm not mistaken, recommendations for cores are no less than the number of virtual machines + host.