RAID is not a panacea or how to improve security?

Question

Backup

Good time, habravchane.
The story is not pleasant, so I'll forgive the help. There was a Linux server on the 5th raid of 4 screws. One fine day, two disks from the raid set sail, the raid could not be raised. The server stored large amounts of data, ~5 TB.
The question is: What can be done in terms of security and fault tolerance, in addition to backups, so that this does not happen again?
Upd: In general, as I thought, the best solution in this situation is backups. Thanks to all.

Answer 1

1) SMART monitoring. If you ever watched what smart tells you, you would have taken action long before such a sudden death.
2) Do not use 5 raid. Bad decision in every way, miser pays twice.
3) Backups (Captain Evidence, sorry, could not resist)
4) When choosing disks for a server, look at such a characteristic as MTBF. Keep in mind that those shitty screws that flooded all the counters are in no way designed for tough 24x7 operation.
5) The screws must have good cooling, they do not like overheating. Ideally, the screws should be in ventilated baskets.

Answer 2

Add more servers with replication in the same (HA) or in another but nearby (DR) data center.

Answer 3

Backups completely solve this problem with minimal financial and labor costs, why crutches?

Answer 4

Collect 10ku, do not save. Look in smart, read mdadm reports (it starts resync if there are broken clusters. And most importantly, it will report a dead disk). Don't use hardware raids. Actively read cache on ssd or in memory if there is a lot of it. Do not use desktop hdd (especially greens). I mean quite desktop. There are those that position themselves as "desktop" but feel good in soho servers - caviar black, for example.
Well, yes - if the data is very important - store it somewhere else. At least even on external drives in another room (most importantly, not in the same place as the server). Fires, robbery… well, we got it.

Answer 5

If several servers look towards DRBD,GlusterFS.