WebSockets not Reliably Routing to Services After Canary Deployment

[Gideon-Felt]

Describe the bug

Using Canary to deploy service that is also scaled using Keda, after Flux reconciles, and Flagger runs through my 25% canary load, once the pods for our services are redeployed as service-primary-xxx web-sockets as handled by socket.io in our node.js service stop being able to initialize. all things being equal, no changes to ingress; just removing the Flaggers Canary manifests from our Flux repository, and those web-socket connections start functioning 100% perfectly.

To Reproduce

• Having already an Nginx-Controller (helm chart), (ours running on AKS) configure an Ingress manifest, and a memory scaling Keda ScaledObject, configured properly according to Flagger documentation, connected to a Deployment servicing a node.js application (nest.js in our case) that uses socket.io to serve web-socket connections. And attempt to make connection with standard Prefix Ingress route.
• Observe once primary images deploy for the service that establishing connections over web-socket doesn't work (503), or in some cases very flaky (1 out of 100, however there were other variables at play when we observed this behavior, once those variables where gone, networking just plain did not work).
• Remove the Flagger manifest and commit to Flux repository.
• After this deployment Observe that web-sockets now routing properly to the intended services.

Expected behavior

Even after a Canary deployment that web-socket traffic will still route to the intended services.

Additional context

• Flagger version: 1.x
• Kubernetes version: 1.29.11
• Service Mesh provider:
• Ingress provider: Nginx