V
V
Vitaly Karasik2021-04-25 19:11:44
Amazon Web Services
Vitaly Karasik, 2021-04-25 19:11:44

How to organize an incremental S3 backup?

The customer asked for a daily backup of objects from S3.
In a couple of minutes I wrote

aws s3 sync s3://origin-bucket/folder s3://backup-bucket/folder- CURRENT_TIMESTAMP

But then I thought about it. The customer dreams
- about something incremental
- and even a zipped archive would be nice (for simplicity, let's say that documents are stored, and I want the ability to roll back to the desired version, "like in Git").

As far as I understand, these wishes are not easy to fulfill - the first is due to the lack of a simple ability to filter objects by date, and the second - because this will require downloading files to a local machine.
I'm thinking of enabling versioning in S3 for backup. Or is there a more elegant solution?

UPDATE: Thanks a lot to everyone who replied! You made me think, and I realized that I should use S3 versioning and replication, instead of reinventing the wheel with scripts and cronjobs.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
R
Romses Panagiotis, 2021-04-25
@vitaly_il1

As a purely "knee" option, use:
EC2 instance with running software:
1. Mount S3 (source, target) as a folder: goofys or s3fs .
2. As backup software: Bup for deduplication. Able like Git...
At the input is the folder of the source bucket, and at the output - the target.
Haven't tried it myself. Just like an idea.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question