How to automate a backup of 1000 databases on several servers?

Question

PostgreSQL

There are about 1000 databases on 2-3 servers.
Each database needs a separate, independent backup: once a week full and incremental every hour.
Each backup must first be encrypted, then put in S3 (well, or sent to another server in any other way).
In parallel, on another server, you need to make recovery attempts and send a success / failure report.
It should be borne in mind that each server, its hundreds of databases, must be backed up sequentially (and not put disks).

If for one database it is still possible to somehow set up such a scheme with difficulty, then for 1000 some kind of automation is needed. How? How?

Answer 1

1000 needs some kind of automation. How? How?

In the simplest version - an ordinary script that would take a list of databases and go through it, archiving each with its own password.
But in general - the meaning of this activity is not very clear. Not out of the blue did you have a thousand bases? If these databases are not yours, that is, some clients, and you are a hoster, nothing prevents you from backing up the entire DBMS in one piece, and if necessary, pulling out only the necessary databases / tables from this backup.

Answer 2

and incremental every hour.

There is no such.
Full-fledged PitR (possibly with diff copies to speed up recovery, but only if you are confident in this third-party mechanism - natively postgresql does not support such an operation) - is removed entirely from the cluster, with all databases at once. Accordingly, revise the requirement for "Each database needs a separate, independent backup."
Under the requirement "Each database needs a separate, independent backup" - only pg_dump, which, accordingly, will make it possible to restore only to this very point in the start time of the dump transaction.

Answer 3

bash script with pg_dump
and that's it.

Answer 4

Ansible