kube-vip
Node labels have the wrong value
The enable_node_labeling feature does add the wrong ip to the node label.
I use the kubevip cloud controller together with kubevip to provide a HA kubernetes api endpoint and IPs for services of type loadbalancer. The kubevip daemonset manifest is created with the following command:
/usr/local/bin/ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:v0.6.4 vip /kube-vip \
manifest \
daemonset \
--interface eth1 \
--address 192.168.1.230 \
--inCluster \
--taint \
--controlplane \
--services \
--arp \
--leaderElection \
--enableNodeLabeling
My cidr-global Range for the cloud controller: 192.168.1.231-192.168.1.239
Expected behavior As soon as a service loadbalancer gets an ip address from the global range, the node should get the label kube-vip.io/has-ip=192.168.1.231. 231 beeing an example from the global range. But the label is always populated with the ip 192.168.1.230, even if the 230 is not on the node.
Environment (please complete the following information):
- OS/Distro: Arch
- Kubernetes Version: v1.28.4+k3s2
- Kube-vip Version: 0.6.4
Kube-vip.yaml:
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/name: kube-vip-ds
app.kubernetes.io/version: v0.6.4
name: kube-vip-ds
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: kube-vip-ds
template:
metadata:
labels:
app.kubernetes.io/name: kube-vip-ds
app.kubernetes.io/version: v0.6.4
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
containers:
- args:
- manager
env:
- name: vip_loglevel
value: "5" # Default 4
- name: vip_arp
value: "true"
- name: port
value: "6443"
- name: vip_interface
#value: eth1
value: eno1
- name: vip_cidr
value: "32"
- name: cp_enable
value: "true"
- name: cp_namespace
value: kube-system
- name: vip_ddns
value: "false"
- name: svc_enable
value: "true"
- name: svc_leasename
value: plndr-svcs-lock
- name: vip_leaderelection
value: "true"
- name: vip_leasename
value: plndr-cp-lock
- name: vip_leaseduration
value: "5"
- name: vip_renewdeadline
value: "3"
- name: vip_retryperiod
value: "1"
- name: enable_node_labeling
value: "true"
- name: address
#value: 192.168.33.9
value: 192.168.1.230
- name: prometheus_server
value: :2112
image: ghcr.io/kube-vip/kube-vip:v0.6.4
imagePullPolicy: Always
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
hostNetwork: true
serviceAccountName: kube-vip
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
updateStrategy: {}
Shortened debug output:
time="2023-12-27T21:01:01Z" level=info msg="(svcs) starting services watcher for all namespaces"
time="2023-12-27T21:01:01Z" level=debug msg="node node03 labels: map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:k3s beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:node03 kubernetes.io/os:linux node-role.kubernetes.io/control-plane:true node-role.kubernetes.io/etcd:true node-role.kubernetes.io/master:true node.kubernetes.io/instance-type:k3s plan.upgrade.cattle.io/server-plan:f7bab2461b130a56013b92448b5d262b9374e133294b1763545c1590]"
time="2023-12-27T21:01:01Z" level=debug msg="setting node label `has-ip=192.168.1.230` on node03"
time="2023-12-27T21:01:01Z" level=debug msg="(svcs) [pihole-dhcp] has been added/modified with addresses [192.168.1.232]"
time="2023-12-27T21:01:01Z" level=debug msg="[STARTING] Service Sync"
time="2023-12-27T21:01:01Z" level=debug msg="init enable service security: false"
time="2023-12-27T21:01:01Z" level=info msg="(svcs) adding VIP [192.168.1.232] for [dns/pihole-dhcp]"
time="2023-12-27T21:01:01Z" level=debug msg="(svcs) broadcasting ARP update for 192.168.1.232 via eno1, every 3000ms"
time="2023-12-27T21:01:01Z" level=debug msg="updated node node03 labels: map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:k3s beta.kubernetes.io/os:linux kube-vip.io/has-ip:192.168.1.230 kubernetes.io/arch:amd64 kubernetes.io/hostname:node03 kubernetes.io/os:linux node-role.kubernetes.io/control-plane:true node-role.kubernetes.io/etcd:true node-role.kubernetes.io/master:true node.kubernetes.io/instance-type:k3s plan.upgrade.cattle.io/server-plan:f7bab2461b130a56013b92448b5d262b9374e133294b1763545c1590]"
time="2023-12-27T21:01:01Z" level=info msg="[service] synchronised in 29ms"
I can't remember who implemented this, but this only applies the IP address of the control plane address as a label.
The feature was introduced in the following pull request: https://github.com/kube-vip/kube-vip/pull/617
My goal is that a pod "follows" a vip. It needs to be on the same node as the vip. Only then I'm able to get the IP from the source client (53/UDP DNS). Is that somehow possible?
Kube-vip is designed so that a vip can follow a pod
https://kube-vip.io/docs/usage/kubernetes-services/?query=external#load-balancing-load-balancers-when-using-arp-mode-yes-you-read-that-correctly-kube-vip-v050
You'll need to enable per service loadBalancing and have a service with externalTrafficPolicyLocal, however if needed we could also update the node labelling so that it updates the labels per node for service IPs, not just the Control Plane VIP.
I tried that but I'm not sure how that works. I've enabled Service Election and configured a LoadBalancer with externalTrafficPolicy: Local. But at the moment the vip is on another node than the pod. Also tested with vip_leaderelection "false" and enableServicesElection "true".
Current env variables:
- name: vip_loglevel
value: "5" # Default 4
- name: vip_arp
value: "true"
- name: port
value: "6443"
- name: vip_interface
value: eno1
- name: vip_cidr
value: "32"
- name: cp_enable
value: "true"
- name: cp_namespace
value: kube-system
- name: vip_ddns
value: "false"
- name: svc_enable
value: "true"
- name: svc_leasename
value: plndr-svcs-lock
- name: vip_leaderelection
value: "false"
- name: vip_leasename
value: plndr-cp-lock
- name: vip_leaseduration
value: "5"
- name: vip_renewdeadline
value: "3"
- name: vip_retryperiod
value: "1"
- name: enable_node_labeling
value: "true"
- name: address
value: 192.168.1.230
- name: prometheus_server
value: :2112
- name: enableServicesElection
value: "true"
Loadbalancer
apiVersion: v1
kind: Service
annotations:
kube-vip.io/loadbalancerIPs: 192.168.1.232
metadata:
name: pihole-dns
namespace: dns
spec:
externalTrafficPolicy: Local
internalTrafficPolicy: Cluster
ports:
- name: dns
nodePort: 30666
port: 53
protocol: TCP
targetPort: dns
- name: dns-udp
nodePort: 30199
port: 53
protocol: UDP
targetPort: dns-udp
selector:
app: pihole
release: pihole
sessionAffinity: None
type: LoadBalancer
Nevermind. I think I found the problem. Used the wrong env variable. It's svc_election and not enableServicesElection. The documentation seems not to be up-to-date on https://kube-vip.io/docs/usage/kubernetes-services/?query=external#load-balancing-load-balancers-when-using-arp-mode-yes-you-read-that-correctly-kube-vip-v050
Using service LoadBalancing fixed my problem. No need for additional node labels. Thank you for the help and idea.