kubernetes-ingress-controller http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred})

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

On brand new install of Kong using Helm Chart Version 2.8.0, getting the following two errors on an Azure Kubernetes Service Cluster, one error in the proxy logs and the other on the ingress logs.

Proxy - http2 requests not supported yet
Ingress - *checking config status failed: %!w(kong.APIError=&{500 An unexpected error occurred})

Subsequently installed it on minikube to make sure that the issue wasn't specific to an AKS cluster and got the same set of errors.

More detail error in Proxy: 3714 [lua] api_helpers.lua:511: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:424: /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: http2 requests not supported yet

Expected Behavior

No errors.

Steps To Reproduce

helm install kong kong/kong -n kong -f .\values.yaml --version 2.8.0

Value file:
env:
  prefix: /kong_prefix/
  database: "off"
  LOG_LEVEL: "error"
ingressController:
  ingressClass: "kong"
  enabled: true
  installCRDs: false

Kong Ingress Controller version

KIC 2.3

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"ba58f86b00f6b0f0b7694a75464aa7806f8bf6fc", GitTreeState:"clean", BuildDate:"2022-03-30T23:40:46Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

Also, getting the same thing using minikube:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

Kong Ingress Controller seems to be working though...so not sure what these errors mean and how concern I should be.

Apr 24 '22 18:04 evhfla

I'm also seeing this on a straightforward installation of the 2.8.0 helm chart.

I found a proxy PR to support http2 on the status endpoint:

https://github.com/Kong/kong/pull/8528

This is an exact output from proxy:

2022/04/25 08:58:07 [error] 1110#0: *1656702 [lua] api_helpers.lua:511: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:424: /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: http2 requests not supported yet
stack traceback:
        [C]: in function 'capture'
        /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:268>

stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/lapis/application.lua:424: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:146: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:183: in function </usr/local/share/lua/5.1/lapis/application.lua:181>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:189: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:1423: in function 'admin_content'
        content_by_lua(nginx-kong.conf:360):2: in main chunk, client: 127.0.0.1, server: kong_admin, request: "GET /status HTTP/2.0", host: "localhost:8444"
127.0.0.1 - - [25/Apr/2022:08:58:07 +0000] "GET /status HTTP/2.0" 500 42 "-" "Go-http-client/2.0"

Kong overall is operating fine, but the log volume is annoying and it's not clear if this issue has any further impact.

Apr 25 '22 09:04 danopia

JFTR, just got hit by this issue aswell. It also happens with chart version 2.7.0, works smooth on 2.6.4.

Apr 25 '22 09:04 rcmorano

It looks like the effect is that this new feature is not operational:

The controller can now detect whether a Kong container has crashed and needs a configuration push.

https://github.com/Kong/kubernetes-ingress-controller/commit/39c385e9b31e3871c2e38b2ea8e12ab328e341ad#diff-d8590e8687e7807da0cf97910b88e0ddbfc670866e0973a15c56597b99df9244R59

workaround

These helm chart values seem to restore the broken feature and remove all the error logs:

# Temporary workaround: disable HTTP2 on admin endpoint
# https://github.com/Kong/kubernetes-ingress-controller/issues/2435
admin:
  tls:
    parameters: []

(It seems like KIC could be hitting /status on the status port instead of the admin port, as the status port already has http2 disabled)

Apr 25 '22 10:04 danopia

@rainest do you know whether the above workaround actually leaves the system in a state that correctly is validating configuration? (If you don't know, I'm not asking you to do the work of checking)

As mentioned in the above PR: "since the /nginx_status location is marked as internal, I don't think it will be possible to fetch it using real network socket"... does that mean the status call we are making doesn't actually check the nginx status, or does that only impact metrics?

Just trying to get an idea of whether I can tell my product owners that the upgrade I just put in place will resolve the stability issues or not ;)

Thanks!!!

Apr 26 '22 14:04 jcam

The workaround (disabling HTTP/2 support on the admin API) is fine. It doesn't meaningfully change the operation of the controller, which makes a separate request for each API call anyway rather than using HTTP/2 pipelining. It may technically degrade Manager performance, but I can't think of pages that are making more than 2-3 admin API requests at once, so I doubt the impact would be noticeable.

https://github.com/Kong/kong/pull/8690 is the PR that's planned to fix this/will allow you to turn HTTP/2 back on.

The status endpoint (as of 2.8) provides both generic NGINX status information (connection count and such) and whether or not configuration is present. We only care about the latter, we just can't fetch it separately, and weren't using the endpoint before.

Apr 26 '22 21:04 rainest

Awesome, thank you for the detail!

Apr 26 '22 21:04 jcam

marking this as blocked on https://github.com/Kong/kong/pull/8690

May 02 '22 12:05 shaneutt

Minor related issue with the error formatting should be corrected by #2436

May 02 '22 17:05 rainest

Blocked until https://github.com/Kong/kong/pull/8690 releases (2.8.1)?

May 31 '22 15:05 scseanchow

I have set ADMIN_LISTEN environment variable to "127.0.0.1:8444 http2 ssl" and it was causing this error.

As a workaround, I had to set ADMIN_LISTEN to "127.0.0.1:8444 ssl" to get rid of these errors.

Jul 13 '22 14:07 jayakumarj85

Blocked until Kong/kong#8690 releases (2.8.1)?

Up, It seem it is released and that it can now be fixed

Aug 25 '22 10:08 altais

It seems that https://github.com/Kong/kong/pull/8690 didn't land in 2.8.1: https://github.com/Kong/kong/commit/c5fd7233ab3e0d910fef31bc3eecd8bdc25fe220.

In that case this will be available in 3.0 (it's already included in 3.0 alpha.1 build: https://github.com/Kong/kong/releases/tag/3.0.0-alpha.1)

Aug 25 '22 10:08 pmalek

Resolving this issue is a no-op on the ingress controller side. When Kong releases the version 3.0, making KIC use Kong 3.0 will fix this issue.

Leaving this issue open for sake of discussion & comments if this solution does not work after 3.0 is out.

Sep 06 '22 15:09 mflendrich

not sure if this is supposed to be active but I'm still seeing this behavior on 3.0 🤔 I'm still getting the same lua exception log bloat for the HTTP2 probes. So I still have to disable the http2 listener config.

Nov 11 '22 18:11 redradrat

@redradrat can you double-check your Deployment?

Deploying https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.7.0/deploy/single/all-in-one-dbless.yaml and adding http2 to the admin listen no longer generates this error for me:

15:44:00-0800 esenin $ kubectl get deploy -n kong ingress-kong -oyaml | grep image:                  
        image: kong:3.0
        image: kong/kubernetes-ingress-controller:2.7.0

15:44:04-0800 esenin $ kubectl get deploy -n kong ingress-kong -oyaml | grep -A1 "name: KONG_ADMIN_LISTEN" 
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl http2

15:44:36-0800 esenin $ kubectl logs -n kong deployments/ingress-kong -c ingress-controller | grep  "checking config status" | wc -l
0

15:44:43-0800 esenin $ kubectl logs -n kong deployments/ingress-kong -c proxy | grep "http2 requests not" | wc -l
0

Nov 11 '22 23:11 rainest

kubernetes-ingress-controller kubernetes-ingress-controller copied to clipboard

http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred})

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Kong Ingress Controller version

Kubernetes version

Anything else?

workaround

kubernetes-ingress-controller
kubernetes-ingress-controller copied to clipboard