can not join additional node to cluster

[daniva6]

Issue report

What version of MicroCeph are you using ?

18.2.4 reef (stable)

Use this section to describe the channel/revision which produces the unexpected behaviour.

I had to forcefully remove a node due to a hardware failure. Afterwards I wanted to join a new node with the previous node name and ip address, unfortunately it run in a timeout. Now I'm getting the following error: Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists I then tried to add a new node with a new ip address and name, unfortunately same result.

What are the steps to reproduce this issue ?

1. install new node with 'sudo snap install microceph --channel reef/stable'
2. create token on a cluster node with 'sudo microceph cluster add '
3. join cluster on new node with 'sudo microceph cluster join '
4. Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists

What happens (observed behaviour) ?

Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists …

What were you expecting to happen ?

new node should have joined the cluster …

Relevant logs, error output, etc.

If it’s considerably long, please paste to https://gist.github.com/ and insert the link here.

Additional comments.

I've realized that in the ceph.conf entry 'mon host = ' only the ip of the new hosts are present, the ip's of the two currently operating nodes are missing …

[daniva6]

following timeout error when trying to add a new node: 'Error: Post "http://control.socket/cluster/control": context deadline exceeded'

[nickwales]

Did you get anywhere with this issue? I have the same problem.

[daniva6]

Did you get anywhere with this issue? I have the same problem.

Hi NIck, unfortunately not. If someone knows how update the 'mon host =' statement with the correct ip's, that might solve the problem.

[UtkarshBhatthere]

The mon host = statement is populated using the mon.host.$hostname config entries from MicroCeph's internal DQlite table. With a little bit of SQL magic you can remove and insert new entries in the table, using which MicroCeph will repopulate the conf file (in a few mins). See config 3 below and the ceph.conf file.

$ sudo microceph cluster sql "select * from config"
+----+----------------------+------------------------------------------+
| id |         key          |                  value                   |
+----+----------------------+------------------------------------------+
| 1  | fsid                 | a307994e-03ed-4122-9ca3-3bb289af9665     |
| 2  | keyring.client.admin | AQAsCBpnPTElMRAA1CRvvsceRWWdm8f/SByOJw== |
| 3* | mon.host.workbook    | 192.168.29.152                           |
| 4  | public_network       | 192.168.29.152/24                        |
+----+----------------------+------------------------------------------+

$ pwd
/var/snap/microceph/current/conf
$ cat ceph.conf 
# # Generated by MicroCeph, DO NOT EDIT.
[global]
run dir = /var/snap/microceph/current/run
fsid = a307994e-03ed-4122-9ca3-3bb289af9665
mon host = 192.168.29.152
public_network = 192.168.29.152/24
auth allow insecure global id reclaim = false
ms bind ipv4 = true
ms bind ipv6 = false

[UtkarshBhatthere]

@daniva6 @nickwales can you please try the above method (read Hack) and see if that solves it for you ?

[UtkarshBhatthere]

@sabaini #446

[daniva6]

@UtkarshBhatthere I applied the hack and was able to remove the non-existing nodes. Unfortunately trying to add a new node results now in a timeout error. The microceph commands seem to work (cluster list, status, disk list) but the ceph command hangs, and the node can not bring up the osds anymore.

Is there a possibility to extract data from the disks? Or to import an osd in another cluster?

[UtkarshBhatthere]

Hey @daniva6 can you please provide a bit more information 1. sudo microceph cluster sql "select * from config", 2. ceph mon dump, and 3. Hostnames and IP address for member nodes to compare what goes where.

[daniva6]

@UtkarshBhatthere I did set up a new ceph cluster and had to delete the old one for space reasons - this was a good test for my backups :)

[fzhan]

@UtkarshBhatthere same issue here, similar scenario to @daniva6

1. sudo microceph cluster sql "select * from config

+----+----------------------+------------------------------------------+
| id |         key          |                  value                   |
+----+----------------------+------------------------------------------+
| 1  | fsid                 | xxx     |
| 2  | keyring.client.admin | xxx |
| 3  | public_network       | 192.168.x.x/24                         |                                            +----+----------------------+------------------------------------------+

2. ceph mon dump

epoch 15
fsid xxx
last_changed 2024-09-04T09:52:24.830192+1000
created 2024-01-24T21:59:54.363795+1100
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:192.168.x.a:3300/0,v1:192.168.x.a:6789/0] mon.1.x
1: [v2:192.168.x.b:3300/0,v1:192.168.x.b:6789/0] mon.2.x
2: [v2:192.168.x.c:3300/0,v1:192.168.x.c:6789/0] mon.3.x
dumped monmap epoch 15

[syncronize-issues-to-jira[bot]]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1172.

This message was autogenerated

[fzhan]

it seems like the token record was not recognised:

Feb 13 10:40:31 3.x microceph.daemon[1464027]: time="2025-02-13T10:40:31+11:00" level=error msg="Unable to complete cluster join request" address="192.168.x.a:7443" error="InternalTokenRecord not found" Feb 13 10:40:31 3.x microceph.daemon[1464027]: time="2025-02-13T10:40:31+11:00" level=error msg="Unable to complete cluster join request" address="192.168.x.b:7443" error="InternalTokenRecord not found"

and a query shows

microceph cluster sql "select * from internal_token_records" +----+------------------------+------------------------------------------------------------------+ | id | name | secret | +----+------------------------+------------------------------------------------------------------+ | 35 | 3.x | 589408a1bee6f8832d24061f67ceb13a3c880199b86a8626ba6ba1bae695e122 |

[fzhan]

After the sqlite update, the conf is now changed to only one of the mon address, and here's the log after reboot:

Feb 13 11:45:34 2.x microceph.mon[4472]: 2025-02-13T11:45:34.118+1100 7f46e091ce80 -1 failed to initialize
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Failed with result 'exit-code'.
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Scheduled restart job, restart counter is at 3.
Feb 13 11:45:34 2.x systemd[1]: Stopped Service for snap application microceph.mon.
Feb 13 11:45:34 2.x systemd[1]: Started Service for snap application microceph.mon.
Feb 13 11:45:34 2.x audit[4533]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.mon pid=4533 comm="ceph-mon" exe="/snap/microceph/1139/bin/ceph-mon" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7f739ce5588d code=0x50000
Feb 13 11:45:34 2.x kernel: audit: type=1326 audit(1739407534.358:150): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.mon pid=4533 comm="ceph-mon" exe="/snap/microceph/1139/bin/ceph-mon" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7f739ce5588d code=0x50000
Feb 13 11:45:34 2.x microceph.mon[4533]: 2025-02-13T11:45:34.370+1100 7f739c629e80 -1 mon.2.x@-1(???) e13 not in monmap and have been in a quorum before; must have been removed
Feb 13 11:45:34 2.x microceph.mon[4533]: 2025-02-13T11:45:34.370+1100 7f739c629e80 -1 mon.2.x@-1(???) e13 commit suicide!
Feb 13 11:45:34 2.x microceph.mon[4533]: 2025-02-13T11:45:34.370+1100 7f739c629e80 -1 failed to initialize
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Failed with result 'exit-code'.
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Scheduled restart job, restart counter is at 4.
Feb 13 11:45:34 2.x systemd[1]: Stopped Service for snap application microceph.mon.
Feb 13 11:45:34 2.x systemd[1]: Started Service for snap application microceph.mon.
Feb 13 11:45:34 2.x audit[4592]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.mon pid=4592 comm="ceph-mon" exe="/snap/microceph/1139/bin/ceph-mon" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7fb72d9fd88d code=0x50000
Feb 13 11:45:34 2.x kernel: audit: type=1326 audit(1739407534.606:151): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.mon pid=4592 comm="ceph-mon" exe="/snap/microceph/1139/bin/ceph-mon" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7fb72d9fd88d code=0x50000
Feb 13 11:45:34 2.x microceph.mon[4592]: 2025-02-13T11:45:34.618+1100 7fb72d1d1e80 -1 mon.2.x@-1(???) e13 not in monmap and have been in a quorum before; must have been removed
Feb 13 11:45:34 2.x microceph.mon[4592]: 2025-02-13T11:45:34.618+1100 7fb72d1d1e80 -1 mon.2.x@-1(???) e13 commit suicide!
Feb 13 11:45:34 2.x microceph.mon[4592]: 2025-02-13T11:45:34.618+1100 7fb72d1d1e80 -1 failed to initialize
Feb 13 11:45:34 2.x systemd[1]: snap.microceph.mon.service: Main process exited, code=exited, status=1/FAILURE

Again, this node must have been stucked in previous conf, and the monmap was not synced between ceph, libceph and microceph.

further to trying to join the cluster with two mon:

Feb 13 11:58:56 2.x microceph.daemon[11755]: time="2025-02-13T11:58:56+11:00" level=error msg="Failed to initiate heartbeat round" address="192.168.*.b:7443" error="Post \"http://control.socket/cluster/internal/heartbeat\": context deadline exceeded"
Feb 13 11:58:56 2.x microceph.daemon[11755]: time="2025-02-13T11:58:56+11:00" level=warning msg="Failed to get status of cluster member with address \"https://10.0.1.1:7443\": Get \"https://10.0.1.1:7443/cluster/1.0/ready\": context deadline exceeded"

After several tries, both node fails to start:

Feb 13 15:26:43 192.168.*.a systemd[1]: snap.microceph.daemon.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 15:26:43 192.168.*.a systemd[1]: snap.microceph.daemon.service: Failed with result 'exit-code'.
Feb 13 15:26:43 192.168.*.a systemd[1]: snap.microceph.daemon.service: Consumed 1.464s CPU time.
Feb 13 15:26:43 192.168.*.a systemd[1]: snap.microceph.daemon.service: Scheduled restart job, restart counter is at 4.
Feb 13 15:26:43 192.168.*.a systemd[1]: Stopped Service for snap application microceph.daemon.
Feb 13 15:26:43 192.168.*.a systemd[1]: snap.microceph.daemon.service: Consumed 1.464s CPU time.
Feb 13 15:26:43 192.168.*.a systemd[1]: Started Service for snap application microceph.daemon.
Feb 13 15:26:45 192.168.*.a microceph.mon[1250]: 2025-02-13T15:26:45.060+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:26:50 192.168.*.a microceph.mon[1250]: 2025-02-13T15:26:50.064+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:26:55 192.168.*.a microceph.mon[1250]: 2025-02-13T15:26:55.065+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:27:00 192.168.*.a microceph.mon[1250]: 2025-02-13T15:27:00.065+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:27:05 192.168.*.a microceph.mon[1250]: 2025-02-13T15:27:05.065+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:27:10 192.168.*.a microceph.mon[1250]: 2025-02-13T15:27:10.065+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
Feb 13 15:27:15 192.168.*.a microceph.mon[1250]: 2025-02-13T15:27:15.065+1100 7fcfc2cdb640 -1 mon.1.x@0(probing) e16 get_health_metrics reporting 3 slow ops, oldest is auth(proto 0 27 bytes epoch 0)
microceph.daemon[3053]: Error: Daemon stopped with error: Daemon failed to start: Failed to re-establish cluster connection: context deadline exceeded

and

microceph status
Error: failed listing disks: Database is still starting

[fzhan]

@UtkarshBhatthere is there a way that I can setup a new cluster, and copy the osd across? Seems like the rockdb has failed to resume.