How to transfer 5 TB of data to the server?

Question

Data transfer

Hello.

There was a task to transfer a large amount of data to the server or to the device to the client.

We have 5 TB of data that will need to be transferred within 12 hours. According to calculations, with an Internet speed of 1 Gbps, this is exactly what will happen. But as I understand it, then fast storage is needed? After all, the HDD will not have a speed of more than 250 Mbps? SSD only if? Or in any case, then more than 12 hours for data transfer will turn out?

And how is it to be conveyed? Is it possible via torrent?

I would be very happy if you could give me an answer.

Answer 1

Is it possible via torrent?

Not necessarily, but an interesting idea.
TeamViewer or AMYY Admin (or whatever this shit is called) you can try, VPN is still there.
Of course, a 1 Gb hard drive will not drag it, but a RAID0 array of 2-3 drives is 115 Mb / s.

Answer 2

In your case, it may be easier to go to the client with a 5+ TB drive.
https://habr.com/ru/company/kingservers/blog/316660/
Added
If the data is deduplicated (ZPAQ, etc.), then it is possible that they can take up significantly less space and then this deduplicated data can be sent via FTPS/HTTP -server.

Answer 3

1Gb is 128MB.
The write speed of an average modern HDD 90 is 150 MB.
So even if the client has a good HDD, the provider will honestly give you the whole Gbps (which is not a fact at all) and this Gbps will remain all the way to the client (which also requires verification), and there will also be no drawdowns during the download, then you get ~ 11 hours 30 minutes.
Which is already very fraught, in case of any unforeseen circumstances.
First of all, you need to increase the channel, after that - check that it really gives out the declared speed and does not sag over time.
After that, already put either RAID (0 or more tricky options) from the HDD, or, if money is not an issue, SSD.

Answer 4

Calculated at 1 Gbps internet speed

More precisely, " up to 1 Gbit".
And therein lies the catch. Not to mention the fact that, upon closer examination, it turns out that this is the maximum theoretically possible speed between the subscriber and the provider's switching point, excluding service traffic.
And then a 2-5-10 Gbit channel to a traffic exchange point between providers, which is divided into all subscribers of the provider of the district-microdistrict-city. Plus, active shapers of zealous subscribers, so that other subscribers do not start to cut off technical support phones.
A similar picture is at the second end (from where to give). And a little better in transit between providers, if we are talking about different cities ...
So to get more speed - it's still "Kamaz floppy disks". Well, or rent a synchronous channel with guaranteed gigabit and sla, which may turn out to be somewhat more expensive than buying KAMAZ, floppy disks and fuel for KAMAZ ....

Answer 5

As an option - deploy your cloud on the source machine. For example owncloud. And to pump out then through the client. The only thing is that you need a domain, a white ip.
This is how partners send terabytes of video to us.

Answer 6

In general, for those who are interested, everything was implemented through an FTP server, the server itself is installed on one device, and a regular FTP client is on the other. There is a possibility of loading. Only until we have achieved the required speed, as the provider somewhere reduces the speed.