SSL protocol error after controller upgrade

[dfateyev]

I have upgraded HAProxy controller from an earlier to the latest version (1.8.3). Unfortunately, an ingress resource for SSL-based backend (actually, the Kubernetes dashboard) stopped working.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  annotations:
    haproxy.org/path-rewrite: /kubernetes-dashboard(/|$)(.*) /\2
    haproxy.org/server-ssl: 'true'
    haproxy.org/check: 'false'
spec:
  ingressClassName: haproxy
  rules:
  - http:
      paths:
      - path: /kubernetes-dashboard
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

Now, I have "502 Bad Gateway" error in HAProxy frontend, and in dashboard logs:

2022/07/21 09:25:51 http2: server connection error from 10.239.96.1:49972: connection error: PROTOCOL_ERROR
2022/07/21 09:25:52 http2: server connection error from 10.239.128.2:51292: connection error: PROTOCOL_ERROR

What can be wrong in this configuration? It worked fine at least for 1.6.x controller versions.

If I remove the "server-ssl" annotation, I get "Client sent an HTTP request to an HTTPS server" valid error as expected.

[dfateyev]

Hi @ivanmatmati, could you check this — it looks like a regression after 1.6.

[ivanmatmati]

Hi @dfateyev , I'll do as soon as possible. Thanks.

[dfateyev]

@ivanmatmati sorry for pinging, did you have a chance to check this issue?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfateyev]

@oktalz please remove the stale label from this issue, thanks!

[dfateyev]

@oktalz I have reproduced it under 1.8.8 and 1.9.0, are there any updates?

[ivanmatmati]

Following the documentation I could reach the UI through Ingress Controller. As it's an HTTP service, you should call it without ssl at all.

[dfateyev]

Following the documentation I could reach the UI through Ingress Controller. As it's an HTTP service, you should call it without ssl at all.

@ivanmatmati could you provide more detail how you managed to do it? Dashboard exposes HTTPS endpoint via Service, not a plain HTTP: see deployment from the link above. Addressing it via HTTP haproxy.org/server-ssl: "false" gives Client sent an HTTP request to an HTTPS server error.

[ivanmatmati]

I just followed the documentation for dashboard installation and then removed the server-ssl annotation.

[dfateyev]

I just followed the documentation for dashboard installation and then removed the server-ssl annotation.

The documentation implies kubectl proxy usage and no ingress. If you ingress what is already proxied by kubectl proxy — then it will be probably work, since kubectl proxy retrieves plain HTTP.

In our case, we want to ingress kubernetes-dashboard service directly, see the message above. It worked fine for 1.6.x, as expected, but stopped to work for later versions, failing with PROTOCOL_ERROR while accessing the endpoint. In general, there is an issue accessing SSL-based endpoints.

[dfateyev]

@ivanmatmati sorry for pinging, but this issue blocks us from controller upgrade to the latest version (we have to keep 1.6 version, where this functionality works). I can provide more details about the issue if needed, although it can be easily reproduced.

[ivanmatmati]

@dfateyev , I can't say much more than I said. For documentation I only talked about installation of dashboard, I don't refer to the rest. So to clarify things:

1. kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml
2. apply the ingress.
3. Reach the dashboard through http frontend of haproxy. First, I reached a blank page because I couldn't get the js files because of mapping with /kubernetes-dashboard, I just replaced it with / and all worked! It was done with latest Ingress Controller code. Please retry with a fresh cluster.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfateyev]

Unstale the issue

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfateyev]

Unstale the issue

[ivanmatmati]

Hi @dfateyev , did you try the thing I've indicated ? We can't keep this issue open if there's no activity on it.

[dfateyev]

did you try the thing I've indicated ? We can't keep this issue open if there's no activity on it.

I will retry, and let you know. Last time it didn't worked, I need to dig it once more.

[dfateyev]

@ivanmatmati I think, I get the reason of the issue. Could you re-check?

$ curl --http1.1 http://<haproxy-ip>/kubernetes-dashboard/
<valid content>

$ curl --http2 http://<haproxy-ip>/kubernetes-dashboard/
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>

2023/02/03 20:52:27 http2: server connection error from 10.239.192.2:60814: connection error: PROTOCOL_ERROR
2023/02/03 20:54:20 http2: server connection error from 10.239.192.2:46358: connection error: PROTOCOL_ERROR

If I revert to controller 1.6.9, HTTP/2 is handled correctly:

$ curl --http1.1 http://<haproxy-ip>/kubernetes-dashboard/
<valid content>

$ curl --http2 http://<haproxy-ip>/kubernetes-dashboard/
<valid content>

upstream Nginx balancer with http/2 support + client browser:
<http/2.0 based valid content>

2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Incoming HTTP/2.0 GET /api/v1/plugin/config request from x.x.x.x:
2023/02/03 21:55:11 Getting application global configuration
2023/02/03 21:55:11 Application configuration {"serverTime":1675461311811}
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Outcoming response to x.x.x.x with 200 status code
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from x.x.x.x:
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Outcoming response to x.x.x.x with 200 status code
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Incoming HTTP/2.0 GET /api/v1/settings/pinner request from x.x.x.x:
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Outcoming response to x.x.x.x with 200 status code
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from x.x.x.x:
2023/02/03 21:55:11 [2023-02-03T21:55:11Z] Outcoming response to x.x.x.x with 200 status code

[dfateyev]

@ivanmatmati have you checked this issue, with new details provided? It looks like the issue can be reproduced with http2-based client after the controller update, but worked for older versions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfateyev]

Please unstale. I believe the case described can be considered.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfateyev]

Unstale the issue

[ivanmatmati]

Hi @dfateyev , I thought I answered you but it's not the case. After some tests, it appears that the question is more related to haproxy. I've observed the same thing as you. The thing is that 1.6.9 uses HAProxy 2.3 while 1.9 uses 2.6. It seems that the difference is at HAProxy level. I've swapped the configuration (with a few adaptations) and it does nothing, so I assume this is not related to the configuration file itself.My best advice would be to report this issue in HAProxy repository with the explanations.

[dfateyev]

@ivanmatmati IMO it's too early to close this ticket: the discussion in escalation isn't finished. BTW, do you have an idea about upgrade options mentioned there? Another question: do we have headers filtering mechanism in the controller now, either for a single ingress resource or for global?

[ivanmatmati]

Hi @dfateyev , I close this issue as there's an other one ongoing in haproxy repository. We can't multiply issues or keep it without activity. If necessary it will be reopen. I'm afraid I don't have any idea about connection upgrade, I'll do some research and keep you informed if any interesting finding. Thanks.