ceph pool is not created when running sunbeam enable observability

[marosg42]

Issue report

What version of MicroCeph are you using ?

reef/beta with sunbeam 2024.1/beta

What are the steps to reproduce this issue ?

Enable several features in sunbeam, usualy enabling observability cause an error

https://bugs.launchpad.net/snap-openstack/+bug/2073734

What happens (observed behaviour) ?

sunbeam enable observability fails on timeout waiting for openstack model. There are ceilometer and gnocchi units in waiting status

ceilometer/0 waiting idle 10.1.166.33 (workload) Leader not ready
ceilometer/1 waiting idle 10.1.1.97 (workload) Leader not ready
ceilometer/2* waiting idle 10.1.97.24 (workload) Not all relations are ready
gnocchi/0 waiting idle 10.1.166.37 (workload) Leader not ready
gnocchi/1* waiting idle 10.1.237.18 (workload) Not all relations are ready
gnocchi/2 waiting idle 10.1.189.155 (workload) Leader not ready

The reason is that ceph pool is not created

unit-microceph-0: 16:33:43 WARNING unit.microceph/0.ceph-relation-changed Error ERANGE: pg_num 64 size 3 for this pool would result in 272 cumulative PGs per OSD (1635 total PG replicas on 6 'in' root OSDs by crush rule) which exceeds the mon_max_pg_per_osd value of 250
unit-microceph-0: 16:33:43 ERROR unit.microceph/0.juju-log ceph:14: Command '['ceph', '--id', 'admin', 'osd', 'pool', 'create', '--pg-num-min=32', 'gnocchi', '64']' returned non-zero exit status 34.
unit-microceph-0: 16:33:43 ERROR unit.microceph/0.juju-log ceph:14: Unexpected error occurred while processing requests: {'api-version': 1, 'ops': [{'op': 'create-pool', 'name': 'gnocchi', 'replicas': 3, 'pg_num': None, 'crush-profile': None, 'app-name': 'rbd', 'compression-algorithm': None, 'compression-mode': None, 'compression-required-ratio': None, 'compression-min-blob-size': None, 'compression-min-blob-size-hdd': None, 'compression-min-blob-size-ssd': None, 'compression-max-blob-size': None, 'compression-max-blob-size-hdd': None, 'compression-max-blob-size-ssd': None, 'group': None, 'max-bytes': None, 'max-objects': None, 'group-namespace': None, 'rbd-mirroring-mode': 'pool', 'weight': 40}], 'request-id': '441182d1e20a95f745487994768a3e33a000eba0'}
unit-microceph-0: 16:33:43 INFO unit.microceph/0.juju-log ceph:14: {"exit-code": 1, "stderr": "Unexpected error occurred while processing requests: {'api-version': 1, 'ops': [{'op': 'create-pool', 'name': 'gnocchi', 'replicas': 3, 'pg_num': None, 'crush-profile': None, 'app-name': 'rbd', 'compression-algorithm': None, 'compression-mode': None, 'compression-required-ratio': None, 'compression-min-blob-size': None, 'compression-min-blob-size-hdd': None, 'compression-min-blob-size-ssd': None, 'compression-max-blob-size': None, 'compression-max-blob-size-hdd': None, 'compression-max-blob-size-ssd': None, 'group': None, 'max-bytes': None, 'max-objects': None, 'group-namespace': None, 'rbd-mirroring-mode': 'pool', 'weight': 40}], 'request-id': '441182d1e20a95f745487994768a3e33a000eba0'}"}

What were you expecting to happen ?

gnocchi pool is created and sunbeam is happy

Additional comments.

As a workaround I use sudo ceph tell mon.* config set mon_max_pg_per_osd 350 …