Load Balancing Servers Not Cleaned Up

[BenB196]

Describe the bug When you add an Ingress via the Ingress Controller, the controller will add the pod's IP(s) as servers under: Traffic Management -> Load Balancing -> Servers. These servers are not deleted when the pod changes IPs. This causes the list of servers to grow massively, even if most of them aren't in use.

To Reproduce

1. Steps
   1. Add an ingress via Ingress controller
   2. Check server list to see pod IP
   3. Delete pod
   4. Let pod recreate
   5. Pod has new IP
   6. Server list contains both old pod IP and new pod IP
2. Version of the Citrix Ingress Controller: 1.13.15
3. Version of MPX/VPX/CPX: VPX - NS13.0 67.39.nc
4. Environment variables (minus secrets)

adcCredentialSecret: nslogin
cicSettings:
  cicConfig:
    NS_COOKIE_VERSION: '1'
    NS_HTTP2_SERVER_SIDE: 'ON'
  required: false
coeConfig:
  distributedTracing:
    enable: false
    samplingrate: 100
  endpoint:
    server: null
  required: false
  timeseries:
    auditlogs:
      enable: false
    events:
      enable: false
    metrics:
      enable: false
      mode: avro
    port: 30002
  transactions:
    enable: false
    port: 30001
crds:
  install: true
  retainOnDelete: false
defaultSSLCertSecret: null
entityPrefix: k8s
exporter:
  image: 'quay.io/citrix/citrix-adc-metrics-exporter:1.4.6'
  ports:
    containerPort: 8888
  pullPolicy: IfNotPresent
  required: false
ignoreNodeExternalIP: false
image: 'quay.io/citrix/citrix-k8s-ingress-controller:1.13.15'
ingressClass:
  - citrix
ipam: true
kubernetesURL: null
license:
  accept: true
logLevel: DEBUG
logProxy: null
namespaceLabels: null
nodeSelector:
  key: null
  value: null
nodeWatch: false
nsIP: x.x.x.x
nsPort: 443
nsProtocol: HTTPS
nsSNIPS: null
nsVIP: x.x.x.x
openshift: false
podIPsforServiceGroupMembers: false
pullPolicy: IfNotPresent
routeLabels: null
serviceAccount:
  create: true
serviceClass: null
setAsDefaultIngressClass: false
updateIngressStatus: true
global:
  systemDefaultRegistry: ''

Expected behavior I would expect the old pod IP to be deleted as it is no longer needed within the VPX, and can create clutter/confusion.

Logs Logs during the deletion + creation of the test pod:

2021-02-04 19:27:45,256 - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:4148] (MainThread) Adjusting application k8s-wordpress-test_80_wordpress-test_svc because of lb service k8s-wordpress-test_http_wordpress-test
2021-02-04 19:27:45,256 - DEBUG - [nitrointerface.py:adjust_service_group:3918] (MainThread) Skipping processing k8s-wordpress-test_http_wordpress-test LB APP for k8s-wordpress-test_80_wordpress-test_svc CS App as it is not meant for me
2021-02-04 19:27:45,257 - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:4148] (MainThread) Adjusting application k8s-wordpress-test_443_wordpress-test because of lb service k8s-wordpress-test_http_wordpress-test
2021-02-04 19:27:45,257 - DEBUG - [nitrointerface.py:adjust_service_group:3920] (MainThread) Processing k8s-wordpress-test_http_wordpress-test LB APP for k8s-wordpress-test_443_wordpress-test CS App
2021-02-04 19:27:45,257 - DEBUG - [globalfunc.py:get_entity_name:93] (MainThread) Entity name generated with k8s-wordpress-test_80 + _SGP_ + k8s-wordpress-test_443_wordpress-test is k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp with length58
2021-02-04 19:27:45,257 - DEBUG - [nitrointerface.py:_create_lbvserver_name:1146] (MainThread) Generated Entity name k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp with str k8s-wordpress-test_80_wordpress-test_svc hash k8s-wordpress-test_443_wordpress-test postfix SGP
2021-02-04 19:27:45,257 - DEBUG - [nitrointerface.py:adjust_service_group:3923] (MainThread) Adjust svcgrp members for k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:45,257 - DEBUG - [nitrointerface.py:_configure_services:1684] (MainThread) configuring service using traditional API (servicegroup:k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp)
2021-02-04 19:27:45,257 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1696] (MainThread) Started services configuration to servicegroup: k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:45,314 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1716] (MainThread) Unbinding 10.42.1.26:80 from servicegroup k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:45,436 - INFO - [nitrointerface.py:_configure_services_nondesired:1721] (MainThread) Unbinding 10.42.1.26:80 from servicegroup k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp is succesful
2021-02-04 19:27:45,436 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1750] (MainThread) Finished services configuration to servicegroup: k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:45,436 - DEBUG - [nitrointerface.py:adjust_service_group:3926] (MainThread) Adjusting Service group members for k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp successful
2021-02-04 19:27:45,436 - DEBUG - [referencetree.py:lookup_tree_node:37] (MainThread) Searching for wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:45,436 - DEBUG - [referencetree.py:lookup_tree_node:43] (MainThread) Node not found for wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:45,436 - DEBUG - [referencetree.py:lookup_regexp_references:151] (MainThread) Lookup for regexp reference markers wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:45,437 - DEBUG - [referencetree.py:lookup_namespace_regexp_references:95] (MainThread) Lookup in namespace regexp reference markers wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:54,200 - DEBUG - [kubernetes.py:configure_cpx_from_endpoints_event:4133] (MainThread) Updating endpoints for k8s-wordpress-test_http_wordpress-test:
2021-02-04 19:27:54,201 - DEBUG - [kubernetes.py:configure_cpx_from_endpoints_event:4134] (MainThread) to-add: [('10.42.1.27', 80)]
2021-02-04 19:27:54,201 - DEBUG - [kubernetes.py:configure_cpx_from_endpoints_event:4135] (MainThread) to-remove: []
2021-02-04 19:27:54,201 - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:4148] (MainThread) Adjusting application k8s-wordpress-test_80_wordpress-test_svc because of lb service k8s-wordpress-test_http_wordpress-test
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:adjust_service_group:3918] (MainThread) Skipping processing k8s-wordpress-test_http_wordpress-test LB APP for k8s-wordpress-test_80_wordpress-test_svc CS App as it is not meant for me
2021-02-04 19:27:54,201 - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:4148] (MainThread) Adjusting application k8s-wordpress-test_443_wordpress-test because of lb service k8s-wordpress-test_http_wordpress-test
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:adjust_service_group:3920] (MainThread) Processing k8s-wordpress-test_http_wordpress-test LB APP for k8s-wordpress-test_443_wordpress-test CS App
2021-02-04 19:27:54,201 - DEBUG - [globalfunc.py:get_entity_name:93] (MainThread) Entity name generated with k8s-wordpress-test_80 + _SGP_ + k8s-wordpress-test_443_wordpress-test is k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp with length58
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:_create_lbvserver_name:1146] (MainThread) Generated Entity name k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp with str k8s-wordpress-test_80_wordpress-test_svc hash k8s-wordpress-test_443_wordpress-test postfix SGP
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:adjust_service_group:3923] (MainThread) Adjust svcgrp members for k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:_configure_services:1684] (MainThread) configuring service using traditional API (servicegroup:k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp)
2021-02-04 19:27:54,201 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1696] (MainThread) Started services configuration to servicegroup: k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:54,222 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1731] (MainThread) Binding 10.42.1.27:80 from servicegroup k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:54,343 - INFO - [nitrointerface.py:_configure_services_nondesired:1735] (MainThread) Binding 10.42.1.27:80 from servicegroup k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp is successful
2021-02-04 19:27:54,343 - DEBUG - [nitrointerface.py:_configure_services_nondesired:1750] (MainThread) Finished services configuration to servicegroup: k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp
2021-02-04 19:27:54,343 - DEBUG - [nitrointerface.py:adjust_service_group:3926] (MainThread) Adjusting Service group members for k8s-wordpress-test_80_sgp_q2n6fic4h6zij6zewacfdxtgwbczkbsp successful
2021-02-04 19:27:54,343 - DEBUG - [referencetree.py:lookup_tree_node:37] (MainThread) Searching for wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:54,344 - DEBUG - [referencetree.py:lookup_tree_node:43] (MainThread) Node not found for wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:54,344 - DEBUG - [referencetree.py:lookup_regexp_references:151] (MainThread) Lookup for regexp reference markers wordpress-test.Endpoints.wordpress-test
2021-02-04 19:27:54,344 - DEBUG - [referencetree.py:lookup_namespace_regexp_references:95] (MainThread) Lookup in namespace regexp reference markers wordpress-test.Endpoints.wordpress-test 

[BenB196]

Anyone got any ideas for this issue?

[kumar-swamy]

This is a known issue due to Citrix ADC behaviour of not deleting the Server Entry when Service group member binding is cleaned up. Since the server entries can be shared across multiple service groups, CIC is not attempting to delete them. But we're aware of the issue and looking for alternate solutions so as to clean up the server entries when they are no longer needed.

[BenB196]

@kumar-swamy thank you for the update.

[mayurmohanpatil]

Hello @BenB196 , Will you please provide your contact details on [email protected] We would like to assist you further for your use case.

Regards, Mayur