I
I
iBird Rose2019-06-08 12:34:09
linux
iBird Rose, 2019-06-08 12:34:09

How to speed up rsync?

There is a folder with about 25k pictures. The pictures are small. It's just a big number.
Task: Move it to another server.
rsync can only scan these files for 12 hours, let alone copy them.
Is there any way to speed up the work?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
I
Igor Deyashkin, 2019-06-08
@iiiBird

In the process of finding out the details in the comments to the question, a solution was found.
In this case, synchronization was performed without using the `--times` option, which saves the file modification time during synchronization. The `--size-only` option was not used, due to which files with the same size are considered the same regardless of the modification time.
And it turns out that the modification dates for all already synchronized files are different, and rsync, when restarted, considers these files to be potentially different and calculates their checksums in order to compare them by content. This took time.
Adding the `--size-only` option has significantly reduced the time it takes to parse files before starting a transfer.
In addition, the `-v` option was used, which displays information about the files being synchronized, which could also slow down the process, although most likely not significantly.

S
Stalker_RED, 2019-06-08
@Stalker_RED

If you don't need to compare which ones are already transferred, then pack everything into archives (tar, gz) and download with the same rsync or scp.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question