How to reallocate space in Ceph?

Question

linux

Help me figure out CEPH

For the test, I deployed CEPH on one machine. I

use directories on the disk as OSD.

Created 7 directories:

/opt/osd1
/opt/osd2
/opt/osd3
...
/opt/osd7

raised rados gateway, as a result, 6 pools turned out:

#ceph osd pool ls
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data

For the test, I set the following parameters (I do not need replication):

osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 30
osd pool default pgp num = 30

During the test, CEPH warned that space was running out on one OSD. I figured adding a new OSD would help and CEPH would redistribute the data on its own (I was wrong!) Now the ceph status is:

~# ceph -s
  cluster:
    id:     3ed5c9c-ec59-4223-9104-65f82103a45d
    health: HEALTH_ERR
            Reduced data availability: 28 pgs stale
            1 slow requests are blocked > 32 sec. Implicated osds 0
            4 stuck requests are blocked > 4096 sec. Implicated osds 1,2,5

  services:
    mon: 1 daemons, quorum Rutherford
    mgr: Ruerfr(active)
    osd: 7 osds: 6 up, 6 in
    rgw: 1 daemon active

  data:
    pools:   6 pools, 180 pgs
    objects: 227 objects, 2.93KiB
    usage:   23.0GiB used, 37.0GiB / 60GiB avail
    pgs:     152 active+clean
             28  stale+active+clean

The space on the OSD has ended and it has gone to the down status:

# ceph osd tree
ID CLASS WEIGHT  TYPE NAME           STATUS REWEIGHT PRI-AFF
-1       0.06857 root default
-3       0.06857     host Rutherford
 0   hdd 0.00980         osd.0           up  0.95001 1.00000
 1   hdd 0.00980         osd.1           up  0.90002 1.00000
 2   hdd 0.00980         osd.2           up  0.90002 1.00000
 3   hdd 0.00980         osd.3         down        0 1.00000
 4   hdd 0.00980         osd.4           up  1.00000 1.00000
 5   hdd 0.00980         osd.5           up  1.00000 1.00000
 6   hdd 0.00980         osd.6           up  1.00000 1.00000

I understand that the problem is in my lack of understanding of how CEPH works, but unfortunately I could not find a solution myself, so I ask for help. Questions that I could not answer:

How to restore CEPH now, there is disk space. I can create an OSD, but how do I get CEPH to redistribute data from one OSD to another?
Why did CEPH write data to only one OSD, I originally created 7 of them?

Answer 1

in general, the author threw off the missing answers and the solution was written for him: https://www.linux.org.ru/forum/general/15813823
again, Size=1, Min Size =1 - atypical and not recommended settings that are not guarantee the correct operation of Ceph Ceph
design in principle assumes Size = 3 , Min Size = 2. Each object is stored in 3 copies, and if two remain, we sound the alarm

Answer 2

And what they show:

ceph osd df tree
df -h

In addition, CEPH tried to write to other OSDs, but could not:

4 stuck requests are blocked > 4096 sec. Implicated osds 1,2,5

We need to look at why there are these blocked requests.
And why 7 folders, and not 10 or 20 at once? (sarcasm)