kubernetes-ingress icon indicating copy to clipboard operation
kubernetes-ingress copied to clipboard
verified publisher icon haproxytech

Loadbalancing algorithm is not working for tcp mode backends

Open benm-stm opened this issue 3 years ago • 5 comments

I use the below helm chart to install haproxy ingress helm install --namespace ingress-controller --create-namespace --set controller.ingressClass='haproxy-pub',controller.ingressClassResource.name='haproxy-pub',controller.kind='DaemonSet',controller.extraArgs={'--configmap-tcp-services=solacescalable/solacescalable-pub-tcp-ingress'} haproxy-pub haproxytech/kubernetes-ingress

apiVersion: v1
data:
  "1883": solacescalable/default-default-1883-pub:1883
  "5671": solacescalable/default-default-5671-pub:5671
  "5672": solacescalable/default-default-5672-pub:5672
  "8000": solacescalable/default-default-8000-pub:8000
  "8443": solacescalable/default-default-8443-pub:8443
  "8883": solacescalable/default-default-8883-pub:8883
  "9000": solacescalable/default-default-9000-pub:9000
  "9443": solacescalable/default-default-9443-pub:9443
kind: ConfigMap
metadata:
  labels:
    app: solacescalable
  name: solacescalable-pub-tcp-ingress
  namespace: solacescalable

i changed the already created configmap to change the loadbalancing to leastconn like below

apiVersion: v1
data:
  load-balance: leastconn
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: haproxy-pub
    meta.helm.sh/release-namespace: ingress-controller
  creationTimestamp: "2022-08-21T16:22:32Z"
  labels:
    app.kubernetes.io/instance: haproxy-pub
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.8.3
    helm.sh/chart: kubernetes-ingress-1.22.4
  name: haproxy-pub-kubernetes-ingress
  namespace: ingress-controller

when i check the haproxy.cfg in the daemonset pod, i find that the loadbalancing is applied to http mode backend but not tcp mode

$kubectl exec -ti -n ingress-controller haproxy-pub-kubernetes-ingress-bwqbx -- cat /etc/haproxy/haproxy.cfg | grep -B2  balance

backend ingress-controller_haproxy-pub-kubernetes-ingress-default-backend_http
  mode http
  balance leastconn
--
  backend solacescalable_default-default-1883-pub_1883
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-5671-pub_5671
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-5672-pub_5672
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-8000-pub_8000
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-8443-pub_8443
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-8883-pub_8883
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-9000-pub_9000
  mode tcp
  balance roundrobin
--
  backend solacescalable_default-default-9443-pub_9443
  mode tcp
  balance roundrobin
--
  backend solacescalable_solacescalable-0_8080
  mode http
  balance leastconn
--
  backend solacescalable_solacescalable-1_8080
  mode http
  balance leastconn
--
  backend solacescalable_solacescalable-2_8080
  mode http
  balance leastconn

I'm i using the right configmap key or is TCP not supported yet ?

benm-stm avatar Aug 22 '22 06:08 benm-stm

Hi @benm-stm, indeed tcp backends are not currently on par with http backends in term of management from ingress controller. That's something I will propose to add. Thanks for reporting.

ivanmatmati avatar Aug 22 '22 07:08 ivanmatmati

As a workaround, i changed the controller default value for loadbalancing to leastconn so that tcp mode get's leastconn by default pkg/annotations/common/main.go

var DefaultValues = map[string]string{
	"auth-realm":             "Protected Content",
	"check":                  "true",
	"cors-allow-origin":      "*",
	"cors-allow-methods":     "*",
	"cors-allow-headers":     "*",
	"cors-max-age":           "5s",
	"cookie-indirect":        "true",
	"cookie-nocache":         "true",
	"cookie-type":            "insert",
	"forwarded-for":          "true",
	"load-balance":           "leastconn",
	"rate-limit-size":        "100k",
	"rate-limit-period":      "1s",
	"rate-limit-status-code": "403",
	"request-capture-len":    "128",
	"ssl-redirect-code":      "302",
	"request-redirect-code":  "302",
	"ssl-redirect-port":      "443",
	"ssl-passthrough":        "false",
	"server-ssl":             "false",
	"scale-server-slots":     "42",
	"client-crt-optional":    "false",
	"tls-alpn":               "h2,http/1.1",
}

I still can inforce balance mode for http mode docker image benmstm/kubernetes-ingress:1.8.3

benm-stm avatar Aug 22 '22 09:08 benm-stm

after applying my workaround i get the right conf:

$kubectl exec -ti -n ingress-controller haproxy-pub-kubernetes-ingress-qzflx -- cat /etc/haproxy/haproxy.cfg | grep -B2  balance
backend ingress-controller_haproxy-pub-kubernetes-ingress-default-backend_http
  mode http
  balance roundrobin
--
  backend solacescalable_default-default-1883-pub_1883
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-5671-pub_5671
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-5672-pub_5672
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-8000-pub_8000
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-8443-pub_8443
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-8883-pub_8883
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-9000-pub_9000
  mode tcp
  balance leastconn
--
  backend solacescalable_default-default-9443-pub_9443
  mode tcp
  balance leastconn

here is my service

kubectl get svc -n ingress-controller haproxy-pub-kubernetes-ingress
NAME                             TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                                                             AGE
haproxy-pub-kubernetes-ingress   NodePort   10.105.152.210   <none>        80:32381/TCP,443:30664/TCP,1024:30362/TCP,1883:30856/TCP,5671:30706/TCP,5672:30819/TCP,8000:31424/TCP,8443:31506/TCP,8883:32594/TCP,9000:31078/TCP,9443:30915/TCP   27m

but when i try to connect using for example the node port 30856, it seems that the loadbalancing algorithm is not applied ! Connections are attributed randomly to the 3 nodes backend Below is the ingress controller pod logs

2022/08/22 15:17:14 INFO    controller.go:166 HAProxy reloaded
[NOTICE]   (267) : Reloading HAProxy
[NOTICE]   (267) : haproxy version is 2.5.8-0cbd0f6
[WARNING]  (267) : config : config: Can't get version of the global server state file '/var/state/haproxy/global'.
[NOTICE]   (267) : New worker (301) forked
[NOTICE]   (267) : Loading success.
[WARNING]  (286) : Proxy healthz stopped (cumulated conns: FE: 188, BE: 0).
[WARNING]  (286) : Proxy http stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy https stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-1883 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-5671 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-5672 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-8000 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-8443 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-8883 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-9000 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (286) : Proxy tcp-9443 stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE]   (267) : haproxy version is 2.5.8-0cbd0f6
[WARNING]  (267) : Former worker (286) exited with code 0 (Exit)
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-1883-pub_1883' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5671-pub_5671/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-5671-pub_5671/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-5671-pub_5671' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-5672-pub_5672' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8000-pub_8000' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8443-pub_8443/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-8443-pub_8443/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8443-pub_8443' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8883-pub_8883/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-8883-pub_8883/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8883-pub_8883' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-9000-pub_9000' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-9443-pub_9443/SRV_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (301) : Server solacescalable_default-default-9443-pub_9443/SRV_2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-9443-pub_9443' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-9443-pub_9443/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-8883-pub_8883/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-8443-pub_8443/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-9443-pub_9443/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-9443-pub_9443' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5671-pub_5671/SRV_3 is UP/READY (leaving forced maintenance).
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-1883-pub_1883' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5671-pub_5671/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-5671-pub_5671' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-5672-pub_5672' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8000-pub_8000' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8443-pub_8443/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8443-pub_8443' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-8883-pub_8883/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-8883-pub_8883' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (301) : backend 'solacescalable_default-default-9000-pub_9000' has no server available!
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_2 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_2 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_2 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-1883-pub_1883/SRV_3 is UP, reason: Layer4 check passed, check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_2 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-8000-pub_8000/SRV_3 is UP, reason: Layer4 check passed, check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-9000-pub_9000/SRV_3 is UP, reason: Layer4 check passed, check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING]  (301) : Server solacescalable_default-default-5672-pub_5672/SRV_3 is UP, reason: Layer4 check passed, check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

i'm i missing something ?

benm-stm avatar Aug 22 '22 15:08 benm-stm

I found out that when i have more than 1 haproxy pod, loadbalancing will act strange. I think that traffic is distributed to all instances of the haproxy pods, thus loadbalancing will not be respected either for http nor for tcp modes

benm-stm avatar Aug 23 '22 11:08 benm-stm

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Sep 23 '22 08:09 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 25 '22 12:10 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Dec 04 '22 03:12 stale[bot]