How to store a lot of data (40+ TB in the future)?

Question

Data storage

Debian 7 x64, fs ext4
Now the data is stored on 2 usb boxes under 4 HDDs (5 raid using the box itself) connected in raid0. There was very little space, we need the same amount at least in the near future. The difficulties are as follows:
- low access speed, everything will be copied to the new storage for a week, or even more
- raid0 - no comments
- even if you buy 3 more boxes and build raid5, then copy everything to them from the current raid0, it doesn't matter how- then I like this solution, then expand the partition with old boxes ... muddy and not reliable
- no one will allocate a budget for a huge storage, somehow you have to do a little bloodshed
Never worked with such a large amount of data, I don’t see a solution yet. Anyone with experience please share.

Answer 1

Buy a mother with 8 or 10 sata ports and a bunch of network cards, set up bonding, assemble raid6 of them (survives the crash of any two disks, fast enough access).
market.yandex.ru/model.xml?modelid=10392262&hid=91020 is the cheapest one.
You won’t get by with a little blood anyway, 10 disks of 4 TB each will cost a pretty penny. Another question is that right now you don’t need 10x4, but when you need it, there will already be 5-6 TB disks.
If we look at the expansion in the future (and, at the same time, on reliability), then I would advise you to stick to this configuration:
every 2 disks are combined in pairs in raid1
from each raid1 pv is made
from all pv is going to vg (for lvm)
pv is cut into pieces of the right size (so as not to keep one file system for 20-40 TB)
When it comes time to expand - free up space to remove one raid1 from pv / vg, do vgreduce (ahtung, long and loads disks), wait, do pvremove , change 2 disks, build a new raid1, make pv out of it, make vgextend.
Well, there you can still use all sorts of USB3 and eSATA for this business, although I would not advise it. If it's too tight and you need more than 10 disks, then there are always PCI-sata (and PCI-express sata) cards, each of which can usually have 4 more disks.
And most importantly - a good PSU, good cooling. And take the discs from different batches.