Microceph low on storage when I barely use 1%

[Gapoly]

Hello,

I have a Docker Swarm with 3 nodes and all Docker's node have a Microceph node on sdb. All nodes have 32GB max of storage and replicate within all 3 of them. I've noticed recently that Microceph isn't replicating anymore when I do ceph -s this is what I get :

  cluster:
    id:     109d1ce5-8e49-4628-bc00-183d62b2d89d
    health: HEALTH_WARN
            mons swarm-node00,swarm-node01,swarm-node02 are low on available space

  services:
    mon: 3 daemons, quorum swarm-node00,swarm-node01,swarm-node02 (age 50m)
    mgr: swarm-node00(active, since 101m), standbys: swarm-node01, swarm-node02
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 3 up (since 101m), 3 in (since 4w)

  data:
    volumes: 1/1 healthy
    pools:   3 pools, 81 pgs
    objects: 27 objects, 699 KiB
    usage:   110 MiB used, 96 GiB / 96 GiB avail
    pgs:     81 active+clean

ceph df
--- RAW STORAGE ---
CLASS    SIZE   AVAIL     USED  RAW USED  %RAW USED
ssd    96 GiB  96 GiB  109 MiB   109 MiB       0.11
TOTAL  96 GiB  96 GiB  109 MiB   109 MiB       0.11

--- POOLS ---
POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr              1    1  577 KiB        2  1.7 MiB      0     30 GiB
cephfs_data       2   64  3.3 KiB        2   24 KiB      0     30 GiB
cephfs_metadata   3   16  129 KiB       23  486 KiB      0     30 GiB

I don't understand why he says low on storage when I don't even use 1%.

[syncronize-issues-to-jira[bot]]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1450.

This message was autogenerated

[sabaini]

Hey there,

mons swarm-node00,swarm-node01,swarm-node02 are low on available space

this is not about OSD disk space. This means the systems the MONs are running on (your swarm nodes) are low on disk on their respective root partitions. ceph df gives you the free disk on the OSDs themselves.

Do you have enough space on the swarm node root partitions?

[Gapoly]

I have plenty of place, 8.8/16GB on my root disk.

[sabaini]

You need to have enough free space on the partition or mount point that the ceph-mon process uses, note that depending on your docker config this could be separate from the root fs.

By default the MON warns if disk goes below 30% free, but this is configurable.

Please refer to upstream docs for details: https://docs.ceph.com/en/squid/rados/operations/health-checks/#mon-disk-low

[Gapoly]

From what I see, I must be using 51% percent of my disk, so 49% is still left. Why do I have that message?

[UtkarshBhatthere]

Are you sure the filesystem is spawning all of the disk (tools like resize2fs may help here) ? Without much details from the system it is hard to suggest any workarounds. Also as @sabaini correctly pointed out, this error is not about the MicroCeph storage (osd space) but disk space on the host machine.