using privateInterface does not seem to pick the right address

[ncopa]

We tries to deploy k0s in linode (akamai). The vpc in linode seems to do source/dest check (and operates in layer 3), which means that kube-router does not work. to work around this we added a vlan (layer 2), which give the node a second interface, eth1.

Kubelet ended up using the ip of eth0, so we tried to use the privateInterface option in the k0sctl config.

For the controller, it picked up the ip address of eth1 as expected, but the worker nodes did not append --node-ip to kubelet, which led to node to node config not working.

Should privateInterface set --node-ip as an extra arg for kubelet?

setting the privateAddress did appear to work.

[kke]

It tries to find an address on the privateInterface that is in not the same as the host's ssh address. If it finds one, it should put it into the PrivateAddress field and act as if one was given.

If a private address is set (via privateAddress, privateInterface or full autodetect) and --enable-cloud-provider is not true, then --node-ip is added to kubelet.

[theosaveliev]

What's the way to fix this?

I have hosts defined like this:

  - role: controller
    ssh:
      address: <public IP>
      <ssh config>
    privateInterface: wg0
    privateAddress: 172.16.100.1

I run k0sctl apply --config k0sctl.yaml I get:

INFO ==> Running phase: Install controllers 
INFO [ssh] <public IP:port>: generate join token for [ssh] <public IP:port> 
INFO [ssh] <public IP:port>: generate join token for [ssh] <public IP:port> 
INFO [ssh] <public IP:port>: validating api connection to https://10.0.0.140:9443 
INFO [ssh] <public IP:port>: validating api connection to https://10.0.0.140:9443 
INFO * Running clean-up for phase: Acquire exclusive host lock 
INFO * Running clean-up for phase: Initialize the k0s cluster 
INFO [ssh] <public IP:port>: cleaning up         
WARN [ssh] <public IP:port>: k0s reset failed    

As you can see, k0s ignored the privateAddress: 172.16.100.1 and it uses 10.0.0.140 instead. That fails because 10.0.0.0/24 network has no connectivity between hosts. I created 172.16.100.0/24 for that purpose.

# ps -ef | grep 10.0.0.140
kube-ap+   22598   22535  4 08:35 ?        00:00:36 /var/lib/k0s/bin/kube-apiserver --enable-bootstrap-token-auth=true --tls-min-version=VersionTLS12 --proxy-client-cert-file=/var/lib/k0s/pki/front-proxy-client.crt --api-audiences=https://kubernetes.default.svc,system:konnectivity-server --requestheader-allowed-names=front-proxy-client --enable-admission-plugins=NodeRestriction --proxy-client-key-file=/var/lib/k0s/pki/front-proxy-client.key --kubelet-certificate-authority=/var/lib/k0s/pki/ca.crt --secure-port=6443 --requestheader-client-ca-file=/var/lib/k0s/pki/front-proxy-ca.crt --egress-selector-config-file=/var/lib/k0s/konnectivity.conf --anonymous-auth=false --authorization-mode=Node,RBAC --client-ca-file=/var/lib/k0s/pki/ca.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --profiling=false --service-cluster-ip-range=10.96.0.0/12 --advertise-address=10.0.0.140 --tls-private-key-file=/var/lib/k0s/pki/server.key --requestheader-username-headers=X-Remote-User --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-issuer=https://kubernetes.default.svc --kubelet-client-key=/var/lib/k0s/pki/apiserver-kubelet-client.key --service-account-key-file=/var/lib/k0s/pki/sa.pub --tls-cert-file=/var/lib/k0s/pki/server.crt --service-account-signing-key-file=/var/lib/k0s/pki/sa.key --v=1 --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --kubelet-client-certificate=/var/lib/k0s/pki/apiserver-kubelet-client.crt --feature-gates= --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --allow-privileged=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/var/lib/k0s/pki/etcd/ca.crt --etcd-certfile=/var/lib/k0s/pki/apiserver-etcd-client.crt --etcd-keyfile=/var/lib/k0s/pki/apiserver-etcd-client.key