kube-vip deletes route if second service with same IP has no backends

[muryoutaisuu]

Describe the bug We had two services with same IP, .spec.type: LoadBalancer and .spec.externalTrafficPolicy: Cluster set. The second service had a selector which did not match anything, hence resulting in endpoint having no backends at all. This caused kube-vip to add the routes of first service, and then immediately delete those routes again when processing the second service with no backends.

Observed output from ip monitor on node:

<ipv6-address> dev lo table 198 proto 248 metric 1024 pref medium
Deleted <ipv6-address> dev lo table 198 proto 248 metric 1024 pref medium

To Reproduce Steps to reproduce the behavior:

1. apply a service with .spec.type: LoadBalancer and .spec.externalTrafficPolicy: Cluster with valid selector
2. apply a second service with same IP, .spec.type: LoadBalancer, .spec.externalTrafficPolicy: Cluster but invalid selector resulting in no backends
3. open shell of a node in second terminal, start ip monitor
4. in first terminal, restart kube-vip daemonset
5. in second terminal, observe the route being added and immediately deleted again:

<ipv6-address> dev lo table 198 proto 248 metric 1024 pref medium
Deleted <ipv6-address> dev lo table 198 proto 248 metric 1024 pref medium

Expected behavior 2 services with same IP, the second service without backends should not delete route of first and valid service.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS/Distro: Ubuntu 20.04
• Kubernetes Version: 1.27.12
• Kube-vip Version: 0.7.1

Kube-vip.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kube-vip
    app.kubernetes.io/version: v0.7.1
  name: kube-vip
  namespace: metallb-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-vip
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-vip
        app.kubernetes.io/version: v0.7.1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: DoesNotExist
            - matchExpressions:
              - key: node-role.kubernetes.io/worker
                operator: Exists
      containers:
      - args:
        - manager
        env:
        - name: vip_arp
          value: "false"
        - name: port
          value: "6443"
        - name: vip_interface
          value: lo
        - name: vip_cidr
          value: "32"
        - name: vip_leaderelection
          value: "false"
        - name: svc_enable
          value: "true"
        - name: vip_routingtable
          value: "true"
        - name: vip_cleanroutingtable
          value: "true"
        - name: prometheus_server
          value: :2112
        - name: svc_election
          value: "false"
        - name: disable_service_updates
          value: "true"
        image: ghcr.io/kube-vip/kube-vip:v0.7.1
        imagePullPolicy: Always
        name: kube-vip
        ports:
        - containerPort: 2112
          hostPort: 2112
          name: metrics
          protocol: TCP
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_ADMIN
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-vip
      serviceAccountName: kube-vip
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

Additional context

first service "a" with backends:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kube-vip.io/loadbalancerIPs: <ipv6-address>
  name: a
  namespace: a
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: <cluster-ipv6-address>
  clusterIPs:
  - <cluster-ipv6-address>
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 32576
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 30224
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    run: haproxy-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: <ipv6-address>

second service "b" with no backends:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kube-vip.io/loadbalancerIPs: <ipv6-address>
  name: b
  namespace: b
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: <cluster-ipv6-address>
  clusterIPs:
  - <cluster-ipv6-address>
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  ipFamilyPolicy: SingleStack
  ports:
  - name: conn-test
    nodePort: 30745
    port: 26384
    protocol: TCP
    targetPort: 26384
  selector:
    app.kubernetes.io/name: netshoot-ipv6
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: <ipv6-address>

ETA: removed metallb stuff, not relevant

[thebsdbox]

Can you get the kube-vip logs with debugging enabled 5.. Also @Cellebyte / @p-strusiewiczsurmacki-mobica don't suppose you'd be able to understand this better than I?

[Cellebyte]

@thebsdbox I will discuss this topic with @p-strusiewiczsurmacki-mobica

[p-strusiewiczsurmacki-mobica]

Hi @muryoutaisuu, @thebsdbox, @Cellebyte I've just created PR on this. It seems that I've made an error when I was implementing route deletion in case of deleted endpoints. Didn't think of the case described in this issue (two services with same loadbalancer IP, but one without endpoints). Should be fixed with #838, as far as I was able to test it.

@muryoutaisuu is it possible that you could build the image from my branch and check if it solves your issue?

[Cellebyte]

@p-strusiewiczsurmacki-mobica I will test it next week when I am back from vacation :)

[muryoutaisuu]

Hi @p-strusiewiczsurmacki-mobica, we've tested and can confirm this solves our issue. Thanks for implementing the fix!