Storage for a large number of medium files?

Question

Network administration

There are several million files, occupying about 1TB. That is, the average file size is about 250Kb. Files are just being added. Adding speed is slow. About 25Mb/day. Reading speed is about 5Mb/s.
Now they are stored on one server and its resources are coming to an end. I want to:
1) get rid of a single point of failure.
2) make the solution scalable by at least 10 times.
3) there are 3 servers for this, on which the processor is heavily loaded, but the disks and memory are free.
4) so ​​that in case of failure of one of the servers, everything is restored by itself.
What options I have considered:
1) ceph. Cephfs is too raw (what they themselves write about). If used as a block device, then you need a 3.4+ kernel (and I have rhel6 - a 2.6 kernel) + the speed of single requests is not very fast there. Plus, they do not get along well with other applications on the same server.
2) drbd. With it, you can get rid of a single point of failure. But with scaling, everything is bad with him. Maximum 3 replicas. In master-master mode, it may require manual intervention when one of the servers crashes to solve split-brain problems. The concept of quorum does not know.
3) glusterfs. There is a feeling that the project is crude, like ceph. I saw a topic on the forum “I have fs stuck in the latest stable version 3.3.1. What to do?" - no response for a week. There were also two posts on Habré from one author about setting up glusterfs. In the second, the author writes that he had to switch to a new version, since the old one hung for unknown reasons and a solution could not be found.
4) iscsi+lvm+gfs2. There are suspicions that in master-master mode, a lot of time can be spent on locks. During tests, listing a newly created small file took 0.5s. As I understand it, this is basically the time of capturing a lock for reading. This option is my favorite so far.
5) cassandra. From what I read about it, I came to the conclusion that this is more of a database than a data warehouse. Tries to keep as much as possible in memory, etc. There is no full recovery - it only happens when reading data, or when manually running a command.
Maybe the respected habra community has more options in mind or has experience that can dispel (or consolidate) my fears?

Answer 1

There is currently no ideal open source solution. If you want reliability and stability - drbd, if you want scalability - glusterfs and ceph. And ceph block devices definitely work on 3.2 kernels, I tried it myself. I really like Glusterfs, this is a very, very, very beautiful product, and it is being developed by Red Hat. But both ceph and glusterfs did not please me with the speed of the “rebuild” after the death and restoration of one of the nodes. True, I tried to use them for virtual storage, in your case this moment may not be so critical. So far I have settled on drbd - fast and stable. Yes, I had to edit split-brain a couple of times, there's nothing wrong with that.

Answer 2

If you are not embarrassed by using the database as a scalable file storage, then you can try mongodb + gridfs + a wrapper in any scripting language.
Related video: video.yandex.ru/users/it-people-ekb/view/56/#

Answer 3

Just sync them with inotify and don't worry, I'm afraid that your workloads and budgets will not allow you to deploy a good cluster solution.