S
S
Sergey Sokolov2013-02-15 03:51:07
linux
Sergey Sokolov, 2013-02-15 03:51:07

How to sync a local folder to Amazon S3?

Wordpress is running on a CentOS server, and it accepts media file uploads to a certain folder. It is necessary to synchronize it into a bouquet on S3, tk. it is from there that the files are distributed to site visitors. The appearance of the uploaded file on S3 is expected almost immediately, because. the link from there it is loaded into the draft post. Therefore cron once-in-5-minutes is not an option, for example.

  1. The first attempt, in a simple way: with the help of FUSE and S3FS, the bouquet was mounted as a device, and the files were loaded directly into it. It turned out to be bad: slow, and there were often errors in processing uploaded files.
  2. The second step: began to keep the local folder into which the download takes place; it was synchronized with lsyncd to a mounted S3 bouquet. Better, but sometimes there are random, subtle bugs and some files don't go through, ending up in a bouquet with zero size. And later, these zero files are never updated to the full size version.

Therefore, now in search of a "third" option. How to synchronize uploads, with a minimum lag, and with concrete reliability, from a local folder to an Amazon bouquet?
Tricks:
  • remote FS mounted S3 bouquet does not work instantly, copying one mp3 may take several. seconds;
  • at the time of synchronization, some files may be open - let's say that someone is directly uploading them to the server right now, or the pictures are reduced and their reduced copies are recorded.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
shadowalone, 2013-02-15
@sergiks

s3cmd try
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX]

P
Paul, 2013-11-15
@PaulIonkin

You can try my project: RioFS - allows you to mount an S3 bouquet as a local folder.
But there are several limitations, one of them is that it does not support "adding" data to files. That is, if you wrote the data to a file and closed it, then in the future you cannot open it and add something to the end.
But usually this is not required. For example, we have a proftpd server that writes files to the RioFS folder (about 10Gb per day of medium-sized files).

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question