cloud-provider-aws icon indicating copy to clipboard operation
cloud-provider-aws copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

Autoscaled K8s Nodes are not added to target group automatically with externaltrafficpolicy as local

Open subhankarc opened this issue 1 year ago • 7 comments

We are trying to understand this behaviour of externalTrafficPolicy attribute in case where the Kubernetes cluster can autoscale in the runtime.

We are using AWS Cluster with "nlb" Load Balancer and have set the "externalTrafficPolicy" attribute of the service to "local".

The experiment is being done to check the "preserving the original source ip address of client" of Istio according to the following documentation, but the problem seems to be the way how Kubernetes service with "externalTrafficPolicy" as Local works along with the NLB load balancers.

azure does not seem to have the problem and also the problem doesn’t exist on earlier version of kubernetes like 1.23.x and 1.24.x

What happened:

When the k8s cluster autoscales and adds a new node to the cluster, we observed that it does not get added to the load balancer target group. We waited for more than an hour but still the target group does not reflect the new node. However, in case of "externalTrafficPolicy" as "Cluster", it gets added to the target group of the load balancer within few mins.

If the pod where the service targets have a new pod autoscaled in the new node, even then it is not added to the target group.

We are of the opinion that this could be a bug and needs to be fixed.

What you expected to happen:

In the first case, even though the node should return healthcheck as failed, the node should get added in the target group.

In the second case, once the autoscaled node has a pod spawned in it, healthcheck also should pass and it should get addedin the target group with successful healthcheck.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): Client Version: v1.21.0 Server Version: v1.25.5
  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): gardenlinux

reference https://github.com/istio/istio/issues/43684

/kind bug

subhankarc avatar Mar 06 '23 11:03 subhankarc