How to build expandable storage from multiple iscsi disks?

Question

linux

My task is to build an easily expandable storage for DICOM PACS images. In fact, you need a regular file cleaner. Resources for this are a couple of NAS from QNAP, on which raid5 is raised and cut into a 5TB iscsi disk. Six months later, another NAS from the same manufacturer will appear in a separate server room, from which another ~ 20TB can be cut off. From all this, you need to create some kind of data warehouse that could be easily expanded. It will be mounted to a single mount point on the PACS server (ubuntu 16.04). I would like the following behavior - if one of the NAS has fallen off, then its files simply become inaccessible, and the files of the remaining NAS are available and new snapshots are written to available iscsi.
I went through a couple of options that did not suit me:

• drive iscsi into JBOD via mdadm and build lvm on top
  
• build lvm on top of iscsi
  

The option did not suit the fact that if one of the NAS is unavailable, all data becomes unavailable or behaves unpredictably.
While I'm working on the option of mounting iscsi devices on the server and combining them using aufs with the write mode on that iscsi, where there is more space. As long as everything is connected, her behavior is perfect. But if one of the iscsi disks becomes unavailable and you try to write something to it, the entire aufs mount point becomes read-only and requires remounting. Does anyone know how this can be overcome? Or perhaps there are other options?

Answer 1

I realized that aufs does not know how to correctly handle the situation when one of the layers is not available (iscsi fell off). In this case, the entire set of layers is marked as faulty and requires iscsi to be remounted and aufs to be recreated. This option doesn't work for me.
I found mergerfs as a replacement. It works similarly to aufs in terms of use (only through fuse, but this is not important for my workloads), but unlike the latter, if one of the mount points is unavailable, mergerfs continues to work normally. It is impossible to read only files from inaccessible iscsi, the rest work fine and are even written ... though there is a nuance. Record point selection policies can be of the following type: where there is less space, where there is more, randomly. And a situation is possible when, according to policy, it is necessary to write to inaccessible iscsi, and then we fall out into honest read only. In principle, not bad, but maybe I'll come up with an option how to get around this limitation. The problem at the moment is that after iscsi has become unavailable, its mount point still knows its free space before the fact of being unavailable. If I find out how to convince the system that if the disk has fallen off, then its size is zero, then I never get into read only, and this is a complete victory. I'll post the result.

Answer 2

I don't know how to solve your problem, because I have no clear idea how the PACS server works.
I will express a crutch decision - to write to the storage chosen by round-robin from the available ones. Is it possible? Or does the system need to be configured to write to a specific location? Is it possible to stream data to some intermediary daemon that will already decide where to write the snapshot?

Answer 3

https://www.mylinuxplace.com/building-raid-over-ne...