Virtual Node DNS service discovery is not working as expected

[PeterK96]

Describe the bug

I have a backend and frontend Deployment with service discovery set to DNS (dns name of the backend Service) for the backend Virtual Node. My backend Deployment has more than two Pods, but if I send GET requests from the frontend to the backend I can only receive responses from two backend Pods. (even though I have more than two backend Pods.)

Steps to reproduce

I started off with the following example: howto-k8s-mtls-file-based. I changed the colorapp in such a way, that it returns the unique pod name and not its color (for testing where the response is coming from). I also modified the blue Deployment to have more than two replicas. After deploying the example and curling the blue backend from within the frontend pod:

curl -H "color_header: blue" color.howto-k8s-mtls-file-based.svc.cluster.local:8080/; echo;

I can only get responses from maximal two blue backend pods even if the blue Deployment has more than two replicas.

Expected outcome

I would expect the backend Kubernetes Service to take over the load balancing entirely and route traffic to all of my backend pods, so if I send GET requests from the frontend to the backend, I can reach every backend Pods.

Environment

• App Mesh controller version v1.10.0
• Envoy version v1.25.1.0-prod
• Kubernetes version 1.24
• Using EKS (yes/no), if so version? yes