U
U
unwrecker2014-10-25 01:29:12
Clustering
unwrecker, 2014-10-25 01:29:12

How to improve cephfs performance?

There is a working ceph cluster of 3 servers: 2 OSD, 3 MON, 2 MDS.
It works great, I must say, but there is a problem: very poor performance of cephfs. I compare the read speed with the RBD partition in the same cluster using fio, this is what I get:
RBD: iops=10709, iodepth=128
cephfs: iops=1087, iodepth=12
When selecting latency ~ 10msec
Well, in practice, everything is very slow: MySQL with databases on a cephfs disk, it barely turns over, operations on many small files are terribly slow, although the linear write speed does not differ from RBD.
OSD nodes are interconnected by a 10-gigabit link.
ceph.conf:

[global]
fsid = e702e3f0-b814-4bd1-98ab-5fb500ab2723
mon_initial_members = srv1, srv2, srv3
mon_host = 10.0.0.1,10.0.0.2,10.0.0.3
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
rbd default format = 2
public_network = 10.0.0.0/24
cluster_network = 10.10.0.0/24

fstab:
10.0.0.1:6789,10.0.0.2:6789,10.0.0.3:6789:/ /ceph ceph name=cephfs,secretfile=/etc/ceph/ceph.client.cephfs.keyring,noatime,rsize=67108864 0 2

Played with rsize - to no avail.
It is clear that, unlike RBD, cephfs has shared access to files that need to be synchronously written, blocked, etc., but why does reading slow down? Yes, and iops 30 times less is somehow too much ...
What can be done?
And another question: mc on the cephfs partition always shows the sizes of directories, which is why (probably) it periodically lags when navigating directories. How would you turn it off?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question