Microceph not starting after upgrade to reef/stable

[usma0118]

Issue report

After upgrading from snap 707 to 975, i am getting error failed to initialize trust store.

I have looked into issue reported #336 and followed copied values to trust store as defined in #269. but still fails.

What version of MicroCeph are you using ?

18.2.0+snap450240f5dd (Single node)

Use this section to describe the channel/revision which produces the unexpected behavior. This information can be fetched from the installed: section of sudo snap info microceph output.

What are the steps to reproduce this issue ?

Upgraded from snap version 707 to 975

What happens (observed behaviour) ?

Mar 01 20:27:02 antaresinc-cluster microceph.daemon[2178]: Error: Unable to start daemon: Daemon failed to start: Failed to initialize trust store: Failed to parse local record "". Found empty certificate
Mar 01 20:27:02 antaresinc-cluster microceph.daemon[2359]: time="2024-03-01T20:27:02+01:00" level=info msg="Daemon stopped"
Mar 01 20:27:02 antaresinc-cluster microceph.daemon[2359]: Error: Unable to start daemon: Daemon failed to start: Failed to initialize trust store: Failed to parse local record "". Found empty certificate
Mar 01 20:27:02 antaresinc-cluster systemd[1]: snap.microceph.daemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 01 20:27:02 antaresinc-cluster systemd[1]: snap.microceph.daemon.service: Failed with result 'exit-code'.
Mar 01 20:27:02 antaresinc-cluster microceph.mds[1408]: starting mds.antaresinc-cluster at
Mar 01 20:27:02 antaresinc-cluster systemd[1]: snap.microceph.daemon.service: Scheduled restart job, restart counter is at 5.
Mar 01 20:27:02 antaresinc-cluster systemd[1]: Stopped snap.microceph.daemon.service - Service for snap application microceph.daemon.
Mar 01 20:27:02 antaresinc-cluster systemd[1]: snap.microceph.daemon.service: Start request repeated too quickly.
Mar 01 20:27:02 antaresinc-cluster systemd[1]: snap.microceph.daemon.service: Failed with result 'exit-code'.
Mar 01 20:27:02 antaresinc-cluster systemd[1]: Failed to start snap.microceph.daemon.service - Service for snap application microceph.daemon.

…

What were you expecting to happen ?

…

Relevant logs, error output, etc.

Mar 03 01:01:59 antaresinc-cluster audit[2422]: AVC apparmor="DENIED" operation="capable" class="cap" profile="snap.microceph.osd" pid=2422 comm="admin_socket" capability=24  capname="sys_resource"
Mar 03 01:01:59 antaresinc-cluster kernel: audit: type=1400 audit(1709424119.298:190): apparmor="DENIED" operation="capable" class="cap" profile="snap.microceph.osd" pid=2422 comm="admin_socket" capability=24  capname="sys_resource"
Mar 03 01:01:59 antaresinc-cluster audit[3590529]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590529 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster audit[3590529]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590529 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster kernel: audit: type=1400 audit(1709424119.438:191): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590529 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster kernel: audit: type=1400 audit(1709424119.438:192): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590529 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster audit[3590542]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590542 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster kernel: audit: type=1400 audit(1709424119.642:193): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590542 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster kernel: audit: type=1400 audit(1709424119.642:194): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590542 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:01:59 antaresinc-cluster audit[3590542]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=3590542 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster audit[1410]: AVC apparmor="DENIED" operation="capable" class="cap" profile="snap.microceph.mon" pid=1410 comm="admin_socket" capability=24  capname="sys_resource"
Mar 03 01:02:00 antaresinc-cluster kernel: audit: type=1400 audit(1709424120.126:195): apparmor="DENIED" operation="capable" class="cap" profile="snap.microceph.mon" pid=1410 comm="admin_socket" capability=24  capname="sys_resource"
Mar 03 01:02:00 antaresinc-cluster audit[3590559]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590559 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster audit[3590559]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590559 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster kernel: audit: type=1400 audit(1709424120.198:196): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590559 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster kernel: audit: type=1400 audit(1709424120.198:197): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590559 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster audit[3590561]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590561 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster audit[3590561]: AVC apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590561 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster kernel: audit: type=1400 audit(1709424120.374:198): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590561 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 01:02:00 antaresinc-cluster kernel: audit: type=1400 audit(1709424120.374:199): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=3590561 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Mar 03 05:01:38 antaresinc-cluster microceph.mgr[1409]: 2024-03-03T05:01:38.591+0100 7f5e60947640 -1 mgr handle_mgr_map I was active but no longer am

Additional comments.

…