kubernetes-ingress Haproxy.cfg is not reloaded when pod IP address changes and standalone-backend annotation is true

We experience a buggy behavior when haproxy.org/standalone-backeng: "true".

It is the most probable cause of error.

When the pod that is under service is deleted and its replicaset or statefulset starts a new one, the haproxy is not reloaded to put new pod IP into backend server list.

We also use ingres with serveral path prefixes, but we use that on different project without issue.

$ /haproxy-ingress-controller --version
2025/02/27 09:55:39 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
HAProxy Ingress Controller v3.0.5 54dded2f.dirty
Build from: github.com/haproxytech/kubernetes-ingress
Git commit date: 2024-12-18T20:53:52Z

$ haproxy -v
HAProxy version 3.0.7-ce35390 2024/12/12 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.7.html
Running on: Linux 5.15.0-1078-azure #87-Ubuntu SMP Wed Dec 18 19:21:34 UTC 2024 x86_64

I am not able to reproduce it clearly on staging environment at the moment.

We decided for standalone backend so the backend-config snippet for each ingress is uniq even though they have the same service in backend. That is necesary to be able to use custom errorfiles on each ingress.

Feb 27 '25 09:02 dosmanak

I was able to get into state, where haproxy.cfg does not correspond with its state.

/ $ grep -A15 ing_customer-test_ing-haproxy-xmldata_jnp /etc/haproxy/haproxy.cfg 
backend ing_customer-test_ing-haproxy-xmldata_jnp
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check maxconn 100
  server SRV_1 10.0.1.61:8080 enabled
  server SRV_2 127.0.0.1:8080 disabled
  server SRV_3 127.0.0.1:8080 disabled
  server SRV_4 127.0.0.1:8080 disabled
  server SRV_5 127.0.0.1:8080 disabled
  server SRV_6 127.0.0.1:8080 disabled
  server SRV_7 127.0.0.1:8080 disabled
  server SRV_8 127.0.0.1:8080 disabled

backend keycloak-dev_svc-keycloak_http
/ $ echo -en "show servers state ing_customer-test_ing-haproxy-xmldata_jnp\r\n"| socat stdio  /run/haproxy-runtime-api.sock 
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
9 ing_customer-test_ing-haproxy-xmldata_jnp 1 SRV_1 127.0.0.1 0 1 1 1 2498 6 3 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 2 SRV_2 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 3 SRV_3 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 4 SRV_4 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 5 SRV_5 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 6 SRV_6 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 7 SRV_7 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 8 SRV_8 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0

Feb 27 '25 12:02 dosmanak

2025/02/27 11:29:17 INFO    service/service.go:169 [transactionID=736de81a-66b0-4993-8d0a-48587b1fe57f] reload required : Service 'customer-test/svc-jnppub': new backend 'ing_customer-test_ing-haproxy-xmldata_jnp'
2025/02/27 11:29:17 INFO    controller/controller.go:203 [transactionID=736de81a-66b0-4993-8d0a-48587b1fe57f] HAProxy reloaded

Feb 27 '25 12:02 dosmanak

I tried to send HUP to haproxy without success.

2025-02-27T12:59:20.617191148Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_1 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617195378Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_2 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617198902Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_3 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617202627Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_4 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617206473Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_5 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617258416Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_6 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617288338Z SIGHUP received, dumping servers states for proxy ing_customer-test_ing-haproxy-xmldata_jnp.
2025-02-27T12:59:20.617300929Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_1 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617304987Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_2 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617308782Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_3 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617312951Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_4 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617316753Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_5 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617320374Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_6 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617324047Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_7 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617327767Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_8 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617331710Z SIGHUP: Proxy ing_customer-test_ing-haproxy-xmldata_jnp has no server available ! Conn: act(FE+BE): 0+0, 0 pend (0 unass), tot(FE+BE): 0+3.
2025-02-27T12:59:20.617274856Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_7 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617389702Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_8 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617394443Z [WARNING]  (30696) : SIGHUP: Proxy ing_customer-test_ing-haproxy-xmldata_jnp has no server available ! Conn: act(FE+BE): 0+0, 0 pend (0 unass), tot(FE+BE): 0+3.

Feb 27 '25 13:02 dosmanak

@dosmanak , Thanks for the complementary input. I'll see that.

Feb 27 '25 13:02 ivanmatmati

We investigated the code and perhaps the controller decides it is not necessary to reload Haproxy. Here https://github.com/haproxytech/kubernetes-ingress/blob/master/pkg/controller/controller.go#L206

Maybe the isNeededReload checks only one haproxy backend that is connected to kubernetes service, but not the other haproxy backend that is created due the standalone-backend annotation.

Feb 27 '25 14:02 dosmanak

Congratulations for helping us in the debugging, actually that's an other part that is concerned about this issue. It has to do with update of servers lists from endpoints events that considers only the regular backend but not the derived ones from standalone annotations. I'll create a fix for that.

Feb 27 '25 16:02 ivanmatmati

Hello. Do you have estimate in what version, the fix will be available? Thank you.

Mar 05 '25 10:03 dosmanak

Hi @dosmanak, it should be available in the next version. This should be around mid june.

Mar 10 '25 10:03 ivanmatmati

That is dissapointing. I will rewrite multiple ingresses into multiple service and I hope that will resove the issue for us.

Mar 12 '25 14:03 dosmanak

I have rewritten it into multiple services and just a single ingress with many host rules. It seems working properly without standalone backend annotation. So no hurry with fix needed.

Mar 15 '25 07:03 dosmanak

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Apr 15 '25 18:04 stale[bot]

/unstale @ivanmatmati any updates?

Apr 17 '25 11:04 dosmanak

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

May 17 '25 12:05 stale[bot]

Seem we have the same kind of issue but without the haproxy.org/standalone-backend annotation but we have a snippet

    haproxy.org/backend-config-snippet: >
      option httpchk

      http-check send meth GET uri /xx/xx/_health ver HTTP/1.1 hdr Host
      xxx

      server xxx xxx
      backup ssl verify none

Sometimes after rollout a statefulset the associated backend has no server.

If I force a reload in the pod with haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $(cat /run/haproxy.pid) -x /var/run/haproxy.sock

The backend start to have server list with all the pods again.

I can't find the root cause

May 19 '25 18:05 mdecalf

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Jun 18 '25 19:06 stale[bot]

Hey, there is another issue perhaps linked to this one. https://github.com/haproxytech/kubernetes-ingress/issues/721

Although if I remember correctly the discussion with @ivanmatmati this issue is thigtly connected to standalone-backends.

Jun 27 '25 06:06 dosmanak

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Jul 27 '25 07:07 stale[bot]