[Question] How does HAProxy update service endpoints?

[evandam]

Hi folks,

We've run into issues with ingress-nginx outlined in https://stackoverflow.com/questions/60982716/how-do-i-reduce-downtime-of-nginx-ingress-route-when-updating-a-destination-pod where ingress-nginx results in requests hanging for 5 seconds during deployments due to pods being terminated and a gap before the respective endpoint object is deleted and Nginx updates its configuration.

The workaround for ingress-nginx seems to be to add a preStop hook to sleep on every container to leave time for the endpoint to be deleted before the pod is terminated, or switch to service-upstream, which has its own issues (mainly relying on kube-proxy, no custom load balancing, etc.).

We're looking into migrating to HAProxy's ingress controller from ingress-nginx and were surprised that HAProxy didn't run into this issue. I was curious how HAProxy updates its backends of a service to prevent this blip, or if there are any other catches to be aware of?

Unrelated, I was also wondering if there was a way to map the SRV_* slots in the statistics to a pod/container/anything on the Kubernetes side? It would be helpful for debugging.

Thanks!

[scalp42]

For the 2nd question, I think the "og" one supports it but not this "version" if I'm correct: https://haproxy-ingress.github.io/docs/configuration/keys/#backend-server-naming

I wonder if that feature could be implemented 🙌

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[scalp42]

Still valid 😅

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oktalz]

hi @evandam ,

when signal that pod is terminating appears, we put that pod in maintenance mode, so pods can close its active connections but will not receive any new connections to it. this is done dynamically through HAProxy runtime, so its basically instant change.

[oktalz]

I was also wondering if there was a way to map the SRV_* slots in the statistics to a pod/container/anything on the Kubernetes side? It would be helpful for debugging.

yes and no :), at first it seems useful, however that would add additional work (and resources) when dynamically changing number of pods. to explain it a bit, if service has 10pods, and you terminate 2, those two are terminated/put to maintenance. but imagine that new one appear, we can just reuse those two that are in maint mode, through runtime, change destination ip/port and immediately start using it without any need for a reload.

So short term it would be useful to have it, but long term it would prevent reusing those.

[oktalz]

@scalp42 documentation you mentioned is related to a different project.

similar behavior can be added (I have nothing against it), but not as a default behavior since long term would be less performant

[evandam]

Got it, thanks for the explanation 👌 Makes sense it's a performance tradeoff. I know it's a different project but I was also curious how the behavior differed between this project.

I'll go ahead and close this out, I don't think we have a need to open a separate issue to support naming backend slots or anything. Thanks!