Unable to add additional worker: CA cert error

[kk2526]

What steps did you take and what happened:

Created elemental cluster with 3 master and 1 worker node. Unable to add additional worker nodes to the cluster now.

K3s-agent logs
Feb 11 16:16:11 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa k3s[1944]: time="2024-02-11T16:16:11Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:54726->127.0.0.1:6444: read: connection reset by peer"

m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa:~ # journalctl -f -u rancher-system-agent.service
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  Creating /usr/local/bin/kubectl symlink to k3s"
[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh"
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh"
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env"
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service"
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  systemd: Enabling k3s-agent unit"
Feb 11 15:13:07 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:07Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service."
Feb 11 15:13:10 m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa rancher-system-agent[1820]: time="2024-02-11T15:13:10Z" level=info msg="[5b4c04217144b97987d14aee1ecbfda5f748cd6c172dac53912c3df32f5b9cc9_0:stdout]: [INFO]  systemd: Starting k3s-agent"`



m-21bb22d7-6117-46b6-b4b4-d06d273a4bfa:~ # curl -v https://127.0.0.1:6444
 Trying 127.0.0.1:6444...
 Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
 ALPN: offers h2,http/1.1
 TLSv1.3 (OUT), TLS handshake, Client hello (1):
 Recv failure: Connection reset by peer
 OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444
 Closing connection 0
 curl: (35) Recv failure: Connection reset by peer`

Anything else you would like to add: Using custom certs

Environment:

• Elemental release version (use cat /etc/os-release): v1.3.5
• Rancher version: v2.7.9
• Kubernetes version (use kubectl version): v1.26.11+k3s2
• Cloud provider or hardware configuration: Harvester: v1.2.1

[kkaempf]

@kk2526 can you provide full details on how to reproduce this issue ?