Allow Calico node ip address manual configuration

[mattirantakomi]

Is your feature request related to a problem? Please describe.

It would be nice to able to set Calico node ip address manually because it is a common practice in many clouds that virtual machine's public ip address is not directly assigned to any interface as there is some kind of one-to-one NAT/firewall and private network between public ip address and virtual machine.

This is a problem when using Calico with Wireguard enabled as it will not advertise itself with correct ip address (public ip) because it detects virtual machine ip address to be some private network address and not that public ip as it is assigned to virtual machine network interface.

Describe the solution you would like

Calico already has configuration option (https://projectcalico.docs.tigera.io/networking/ip-autodetection#manually-configure-ip-address-and-subnet) which allows user to set IP/IP6 environment variables to define ip address manually. There is even some draft on https://github.com/k0sproject/k0s/issues/479#issuecomment-749076445 how to allow user to specify environment variables to Calico deployment on k0s configuration.

Describe alternatives you've considered

No response

Additional context

No response

[mattirantakomi]

Just realized that it couldn't work the way I described as node ip address is a node specific property. Better solution would be IP_AUTODETECTION_METHOD which checks public ip address from external source, eg. ip.jes.fi and uses it as Wireguard endpoint address.

[mattirantakomi]

I opened new Calico feature request: https://github.com/projectcalico/calico/issues/5344

[mikhail-sakhnov]

I am not sure I fully understand the requirement but does setting k0s .Spec.Network.Calico.IPAutodetectionMethod field help you anyhow?

https://docs.k0sproject.io/v1.23.1+k0s.0/configuration/#specnetworkcalico

[mattirantakomi]

@soider It does not help because there is no autodetection method available which could lookup the real public ip address from external source (eg. curl -s https://ip.jes.fi). Autodetection methods first-found/can-reach/interface could only determine ip address which is bound to some network interface but the problem is that the public ip is not directly bound to any interface on k0s node.

[mikhail-sakhnov]

@mattirantakomi

Relying on external service to check IP address sounds error-prone to me.

Will for example kubelet extra arg --node-ip help you combining with setting ip autodetection to empty value (https://projectcalico.docs.tigera.io/networking/ip-autodetection)?

If the node-ip works, then instead of coupling k0s with any external service on the code level, you can just use that service as a part of shell subcommand while building arguments for the k0s binary.

E.g. something like

sudo k0s worker --kubelet-extra-args="--node-ip=`curl -s https://ip.jes.fi `"

[mattirantakomi]

I will try if using --kubelet-extra-args to set node ip works for me.

I don't see that using external service to check public ip is more riskful than can-reach method is already doing:

https://projectcalico.docs.tigera.io/reference/node/configuration#can-reachdestination

The can-reach method uses your local routing to determine which IP address will be used to reach the supplied destination. Both IP addresses and domain names may be used.

Another reliable way to lookup own public ip address is simple dns lookup: dig +short ch txt whoami.cloudflare @1.1.1.1

If that doesn't work, it is likely that the internet connection is not working at all or has serious problems.

[mattirantakomi]

E.g. something like

sudo k0s worker --kubelet-extra-args="--node-ip=`curl -s https://ip.jes.fi `"

Using --node-ip in k0s install command line k0s install controller --enable-worker --kubelet-extra-args="--node-ip=`curl -s https://ip.jes.fi `" does not work as it seems that node ip must be assigned to one of network interfaces:

Dec 29 19:04:15 ubuntu k0s[734]: time="2021-12-29 19:04:15" level=info msg="E1229 19:04:15.310896 1103 kubelet_node_status.go:586] \"Failed to set some node status fields\" err=\"failed to validate nodeIP: node IP: \\\"65.108.12.148\\\" not found in the host's network interfaces\" node=\"ubuntu\"" component=kubelet

[mattirantakomi]

Dirty workaround to get Calico node to pick correct public ip address by assigning it to dummy0 interface on virtual machine: https://github.com/mattirantakomi/dummyip

To be used with k0s cluster configuration:

apiVersion: k0s.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s
spec:
...
  network:
    ...
    provider: calico
    calico: 
      mode: vxlan
      overlay: CrossSubnet
      wireguard: true
      ipAutodetectionMethod: "interface=dummy.*,eth.*,enp.*,ens.*"

[mikhail-sakhnov]

@mattirantakomi

does using cloud-provider integration help you with that?

if you install k0s with arguments for cloud provider like that

sudo ./k0s install controller -c /home/ubuntu/config/k0s.yaml --enable-k0s-cloud-provider=true --enable-worker --enable-cloud-provider

and than annotate node with address

./k0s kc annotate node nodename k0sproject.io/node-ip-external=nodeaddr

the given address will be reconciled as external node ip

[mattirantakomi]

Just tried using k0sproject.io/node-ip-external=nodeaddr but it does not work because Calico node still detects node ip by reading it from node interfaces:

ipAutodetectionMethod: "interface=dummy.*,eth.*,enp.*,ens.*"

It is also possible to specify Calido node ip manually by setting IP environment variable but that option cannot be used in this case because calico-node is running as daemonset so IP env cannot be set individually for every node. As far is I know, there is no ipAutodetectionMethod available which could read node ip address directly from k0sproject.io/node-ip-external annotation.

https://projectcalico.docs.tigera.io/networking/ip-autodetection

[github-actions[bot]]

The issue is marked as stale since no activity has been recorded in 30 days

[mattirantakomi]

I don't think that the issue is completed as it still isn't possible to set calico node ip address manually or detect public ip with external lookup.