Networking for a Ceph cluster?

Question

Storage Area Network

There was a need to raise the Ceph cluster to distribute files to several servers (now 3, but soon there will be up to 5 - 10): forwarding to docker, storing databases, static files, possibly virtual machines.
Given: pfsense router (1 Gbps ports), D-Link DS-1024 unmanaged switch (24 x 1 Gbps), a small opportunity to buy something.
Question: is it possible to implement everything with the network equipment that is available, but without virtual machines? If not, what do you need to buy?
As I understand it, you can do without optics (it is painfully expensive), but simply do link aggregation, but due to the fact that only Ceph will require 2 pcs. VLAN + main network, managed switch still have to buy?

Answer 1

Gigabit is enough for the client network. But for a storage network, at least 10 gigabits is very desirable. Otherwise, there will be terrible lags on the network. Do not even think about vlan! Two independent adapters, well, or one for 10 gigabit, divided into two srio into vlans! We have a cluster in general on infiniband 56gi, and sometimes there is not enough bandwidth.
And do not spare money for the network, it is always not enough! Skychas for 100 kilo rubles you can buy cards and a switch for 10 gigabytes. And optics is not needed if everything is in one server room. Yes, and optics sometimes come out cheaper than copper! She's not expensive at all!