When configured node-ip with an IP different from the default IP of the node IPv6 not configured as InternalIP

[rbrtbnfgl]

trafficstars

Environmental Info: K3s Version:

v1.24.2+k3s1

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Configuring a node with a dualstack node-ip and the IPv4 configured is not the one used as default.

Describe the bug:

When the node is started the configured internalIP is not dualstack but shows only the IPv4

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.addresses[?(@.type == "InternalIP")].address}{"\n"}{end}'

If disable-network-policy is not configured the lack of the IPv6 address will lead to this error

level=fatal msg="host IPv6 unknown"

It should be related to this #5922

The issue is related to kubelet that configures the IPv4 when cloud-provider=external and node-ip is configured (cloud-provider should be removed on kubelet 1.24 but it seems that it's still there) https://github.com/k3s-io/k3s/blob/master/pkg/daemons/agent/agent_linux.go#L126 Configuring node-ip with the dualstack config will not solve the issue because kubelet will not start because dualstack node-ip is not supported when cloud-provider=external. Removing node-ip could solve it but kubelet will use the default IP and not the one configured on K3s. The solution could be removing the cloud-provider flag because it should be already deprecated on kubelet 1.24.

Steps To Reproduce:

• Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

[manuelbuil]

After some investigation, some unorganized facts:

• cloud-provider=external is a requirement in k3s to pick up the ExternalAddresses. Kubelet does not have a way to set this
• If we are in an env with only one interface, things work because k3s is not setting the kubelet's node-ip flag
• If we are in an env with more than one interface, but we are using the main interface, things work because k3s is not setting the kubelet's node-ip flag
• The problem appears when we are in an env with more than one interface and we are not using the main interface. In that case, k3s is setting kubelet's node-ip flag
• kubelet's node-ip flag takes precedence over what the cloud provider collects starting with 1.24. That's why we don't see this problem in previous versions
• If we don't pass the kubelet's node-ip flag, kubelet will consume the IP of the main interface, i.e. it will ignore the --node-ip parameter of k3s
• If we pass a dual-stack value to kubelet's node-ip flag, k3s fails with: kubelet exited: failed to run Kubelet: dual-stack --node-ip \"10.10.10.100,a11:decf:c0ff:ee::10\" not supported when using a cloud provider"
• When kubelet detects a node-ip flag, it sets an annotation https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L95-L113 in the node's object which is read by the k8s' cloud-provider code. If it exists, then it ignores whatever internalAddress the k3s cloud-provider returns

[manuelbuil]

Docs are updated with the workaround to avoid this upstream bug

[cwayne18]

@manuelbuil should we close this out or are there fixes we need to do? If it's an upstream bug and we documented the workaround I think we should close it

[manuelbuil]

we can close it