bug: Plugin: openid-connect: Missing Header X-Forwarded-Proto: https - Regression?

[freeo]

Current Behavior

Ory Hydra (auth server) reports this error:

X-Forwarded-Proto header is missing

And apisix reports this:

2024/07/04 10:33:42 [error] 50#50: *428 [lua] openid-connect.lua:503: phase_func(): OIDC introspection failed: [...] body={"error":"error","error_description":"The error is unrecognizable"}

This must be some kind of regression, because this official video uses Ory Hydra with Apisix and I basically copied the ApisixRoute manifest from this video: https://www.youtube.com/watch?v=SqzS_CULxHQ&t=1203s

I tried this with Apisix 3.9.1 plugin openid-connect and pulling this plugin from master because I saw this new object introspection_addon_headers which I hoped might help me inject the X-Forwarded-Proto header. I still can't rule out if I'm doing anything wrong (wrong usage of Apisix), but one thing is for sure: This headers value is http and therefore Ory Hydra refuses to work with it at all. 2 years ago this worked just fine (the video is 2 years old). I tried all kinds of configs to run Hydra in HTTP mode but to no avail.

I validated the missing header as the core issue, because when I installed the plugin from master as custom plugin and hardcoded this:

525:  core.request.set_header(ctx, "X-Forwarded-Proto", "https")

then Ory Hydra worked just fine with the mentioned config.

Of course this hackery isn't secure, so I need a proper solution for the openid-connect plugin.

Am I missing something here, or is this a regression in the plugin? Or a result of security hardening? How should I actually get this header into the introspection call? Considering I configured introspection_endpoint: http://..., is it correct that X-Forwarded-Proto: http and not https? Must I setup Ory Hydra with its own TLS? My assumption is that Apisix TLS termination should be enough security-wise.

Some background: Running k3s on a local VM with a self signed cert. TLS termination at Apisix works just fine.

Expected Behavior

2024/07/04 18:37:28 [debug] 50#50: *261 [lua] openid-connect-wip.lua:425: introspect(): token validate successfully by introspection

Error Logs

Ory Hydra full output: Could not serve http connection audience=application error=map[message:X-Forwarded-Proto header is missing] p content-length:666 content-type:application/json user-agent:OpenAPI-Generator/1.0.0/go x-forwarded-for:10.42.0.1 x-forwarded-host:hydra.admin x-forwarded-port:9080 x-forwarded-proto:http x-real-ip:10.42.0.1 host:hydra.admin method:POST path:/admin/clients query: remote:10.42.0.160:58634 scheme:http service_name=ORY Hydra service_version=v1.10.7

Steps to Reproduce

1. Install Ory Hydra

image:
  tag: latest-sqlite

hydra:
  config:
    dev: true
    dsn: memory
    ttl:
      access_token: 8h
    urls:
      self:
        issuer: https://hydra-public.default.svc.cluster.local/
      login: http://wrong-on-purpose/login
      consent: http://wrong-on-purpose/conset
    secrets:
      system:
        - QTZvT09fuUYFSDUf0sudf09FmhvSFRWFY=

    log:
      level: debug

2. Configure ApisixRoute with openid-connect to secure any endpoint:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: myapp
  namespace: default
spec:
  http:
    - name: myapp-route
      match:
        hosts:
          - myapp.cluster.local
        paths:
          - /*
      backends:
        - serviceName: myapp
          servicePort: 8080
      plugins:
        - name: openid-connect
          enable: true
          config:
            client_id: 989b6457-f5a7-422f-9f4d-7ab7b0d0f28a
            client_secret: JnyoxL0Qs9LKSsV09aDi5KRT.q
            discovery: http://hydra-public.default.svc.cluster.local:4444/.well-known/openid-configuration
            introspection_endpoint: http://hydra-admin.default.svc.cluster.local:4445/oauth2/introspect
            scope: openid
            bearer_only: true
            realm: master
            introspection_endpoint_auth_method: client_secret_post
            introspection_addon_headers:
              - X-Forwarded-Proto
            redirect_uri: https://httpbin.org/get

3. Try to request with curl, no need for a correct token, it doesn't get that far. Hydra refuses to work with the request because of the missing "X-Forwarded-Proto: https" Header.

Environment

Running on k3s with Apisix Helm Chart 2.8.0, so Apisix 3.9.1

• APISIX version (run apisix version): /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version 3.9.1
• Operating system (run uname -a): Linux apisix-7d7f7b97cd-8888p 5.15.0-91-generic 101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.25.3.1 built by gcc 10.2.1 20210110 (Debian 10.2.1-6) built with OpenSSL 3.2.0 23 Nov 2023
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):