What system to use for lunix backups from s3?

Question

linux

Tell me, please, what do you back up in practice?
Of the features we have a lunix server in which you need to backup folders and mysql.
I would like to have alerts in telegrams and emails.
The system should store backups on s3 and have a client-server architecture.

tried several solutions - we constantly run into some kind of crutch.
of the most interesting - we set up bacula \ bareos and duplicati for ourselves. The first one is too confusing, the second one is fine, but there is no server in fact, and there are a lot of backup machines and I would like to see the big picture.

Thank you!

Answer 1

Backup locally, check performance, send to S3. At any of these stages or all together, you can add monitoring metrics - and thus see the big picture and quickly identify problems.

Answer 2

point to a solution?

I won’t be able to, because I don’t know ready-made systems out of the box to make backups, and deduplication was adequate, and there would also be control over the result.

lyrics

возможности и выбираемый инструмент полностью зависят от того как именно у тебя построены рабочие машины, какая операционная система, где и как хранишь файлы.
К примеру системный раздел windows можно адекватно бакапить только средствами копирования разделов (так как для их работу необходимо корректно восстанавливать не только acl но и hard- и symbolic- links (подскажите мне если знаете такой что сможет работать с бакапом на уровне файлов, кроме штатного от майкрософта но управляемость бакапами там нулевая, плюс почти никакой дедупликации, ну их нафиг)
И главное, никакая из известных мне систем резервного копирования не проводит полностью контроль над бакапами в том виде который требует идеология, а именно - возможность восстановить рабочую систему из бакапа. Максимум что могут сделать - проверить контрольную сумму результата (что само собой не спасет от ошибок в конфиг файлах). И это логично, так как собственно сам факт проверки работоспособности автоматизировать универсально невозможно.
Но вот в частных случаях, когда интегратору/админу известна система от и до, возможно построение своих скриптов автоматизации и формировании отчетов о тестировании среды.

In fact, at one time I used windows bat scripts (ready-made ones were written for win7 and no longer work in win10, but the idea is simple there, everything is easily rewritten) when a snapshot was created on the target disk, then a full copy of the specified directories was created using rsync, but with an indication of the previous version of the replica, for unchanged files, the utility created a hardlink to the previous copy, as a result, an ideal incremental copy is obtained, with which you can work like with ordinary files (extraction from compressed archives and sequential acceptance of increment patches are not required), and to any saved date, if necessary, you can easily delete the old or even any replica, thanks to the correct processing of the hardlink by the file system.
The disk itself can be connected over the network, either by means of windows (.vhd files) or iscsi
Compression - supported, but only weak ntfs.
The disadvantage of the approach is that at the time of creating a backup, full write access to the entire archive is required, i.e. this is a terrible operation, if something goes wrong, you can lose all copies. If you close your eyes to rootkits that break file systems bypassing the OS, directly writing disks (an unlikely scenario that you need to protect yourself from even before thinking about backups), then you can protect yourself by setting access rights to old replicas, there are console utilities for everything
The second drawback is that a year of daily backups with even an average number of files in the final partition will create billions of hardlinks (the limit on the number of hardlinks per file is 1024, which is about three years of daily backups), and for example, checking such a disk with chkdsk may take days and weeks.
----------------------------------------
Another approach if you want to create copies of shared shares on server (linux of course), you choose cow file systems, with support for online snapshots - btrfs/zfs. Every day or even every hour you create a snapshot and store it for as long as necessary, deleting the ones you don’t need when you want (the operation is instant, does not depend on the amount of data). This will protect data from client-side problems, such as erroneous deletions or ransomware, etc.
To protect data from problems with server hardware, for example, you can use the btrfs incremental backup feature , when the difference between two specified snapshots (previous and current) is sent to a remote server, packing it into one file (more precisely, a stream). those. these patches can be stored and applied to a pre-saved startup state of the file system. Of course, the remote server must also store snapshots (by the way, patches can be stored on a third server).
Those. as a result, the same bonuses as in the first approach, but it requires a full-fledged backup of a remote server that receives and deploys snapshot patches.
Compression is also supported, and you can use the best - zstd
In both cases, I advise you to create automatic reports on changes (not only general ones, but for example, by users, how many changes which users made, in graphs), i.e. even the snapshot size information is more than useful, if you normally get X GB diffs weekly, and today all of a sudden you get 0 or 100X it's already a cause for concern.
----------------------------------------
Checking backups for recoverability.
There must be a machine on the network that does only this - it takes turns raising the backups of each machine individually or immediately of the entire infrastructure on virtual machines, starts services and databases and conducts some kind of checks. Unfortunately, the creation of adequate automated tests of the entire infrastructure is too complicated a task and few people bother with it, moreover, automatically deploying a working system from backups and putting it into operation is already a feat, so someone will have to do this in a semi-manual mode (i.e. service provides file recovery, and a person needs to run and see if, in principle, everything works as expected)

Answer 3

Look towards Veaam, though I don’t know if he can work with S3.