How to quickly copy 40 million files to another server?

Question

linux

There is a server with pictures
on it 40 million files, there are small ones of 5-10 kb, there are more 50-100 then the options are not to do it by files, but it can immediately rewrite the entire disk image or copy it by sectors. I would like to be able to do this in a few hours.

Answer 1

Something like this:
On an aggregated channel 2x1Gbit gives out 100 megabytes per second. If you cheat on the block sizes bs= and mtu, and also turn off encryption for ssh, you can, I think, squeeze out all 200 megabytes.
Conduct, on unmounted! (or readOnly mounted) disk!
No wonder dd stands for DestroyData.
Options for setting encryption (-c) and speed (better - more):

Answer 2

I migrated via zfs send | zfs recv via ssh. The speed was only limited by the read/write speed of the HDD.

Answer 3

tar -cC source-dir . | ssh target-ip "tar -xC target-dir"

Answer 4

An overview of different ways to quickly copy:
moo.nac.uci.edu/~hjm/HOWTO_move_data.html

Answer 5

rsync across directories via GNU parallel.

Answer 6

The main problem is how the files are arranged in the directory tree. Often the gag is related to building a list of files, and not to processing it.

Answer 7

if the interserver network is fully controlled and there is no one else in it, then netcat can be used instead of ssh. minus encryption == plus speed/resources