Peer Discovery with DNS record wont cluster

[womblep]

Describe the bug

I am upgrading from 3.11 to 3.13 and my working peer discovery wont work in the new version. It discovers the A records but wont do anything with them.

| 1716362235833 | {"time":"2024-05-22 07:17:15.833049+00:00","level":"info","msg":"Addresses discovered via A records of peernodes.mq-nonpci.apse2.lab.securecall.local: 10.10.17.226, 10.10.23.236","pid":"<0.253.0>","domain":"rabbitmq"}
| 1716362235834 | {"time":"2024-05-22 07:17:15.834162+00:00","level":"info","msg":"Addresses discovered via AAAA records of peernodes.mq-nonpci.apse2.lab.securecall.local: ","pid":"<0.253.0>","domain":"rabbitmq"}
| 1716362237520 | =PROGRESS REPORT==== 22-May-2024::07:17:17.511731 ===
| 1716362237520 | supervisor: {local,inet_gethost_native_sup}
| 1716362237520 | started: [{pid,<0.97.0>},{mfa,{inet_gethost_native,init,[[]]}}]
| 1716362237521 | =PROGRESS REPORT==== 22-May-2024::07:17:17.520454 ===
| 1716362237521 | supervisor: {local,kernel_safe_sup}
| 1716362237521 | started: [{pid,<0.96.0>},
| 1716362237521 | {id,inet_gethost_native_sup},
| 1716362237521 | {mfargs,{inet_gethost_native,start_link,[]}},
| 1716362237521 | {restart_type,temporary},
| 1716362237521 | {significant,false},
| 1716362237521 | {shutdown,1000},
| 1716362237521 | {child_type,worker}]
| 1716362237609 | {"time":"2024-05-22 07:17:17.608862+00:00","level":"error","msg":"Peer discovery: could not discover and join another node; proceeding as a standalone node","line":245,"pid":"<0.253.0>","file":"rabbit_peer_discovery.erl","domain":"rabbitmq.peer_discovery","mfa":["rabbit_peer_discovery","retry_sync_desired_cluster",3]}

Reproduction steps

cluster_formation.peer_discovery_backend = dns cluster_formation.dns.hostname = cluster_formation.discovery_retry_interval = 500 cluster_formation.discovery_retry_limit = 10 cluster_partition_handling = autoheal

Expected behavior

The IP addresses found would be used as peers

Additional context

This was working in 3.11 I restarted the whole cluster thinking it was an upgrade issue but they still wont find each other