remoteJWKS needs a caCert option

[vacan1t]

Description: When setting up JWT authentication we need to be able to set custom CA-certificate to allow Envoy to trust our internal HTTPS JWKS server.

Snippet from SecurityPolicy:

  jwt:
    providers:
    - name: poc-oidc
      remoteJWKS:
        uri: https://poc-oidc.internal.domain.com/keys

Relevant debug logs:

[2024-06-04 13:15:36.191][1][debug][connection] [source/extensions/transport_sockets/tls/cert_validator/default_validator.cc:325] verify cert failed: X509_verify_cert: certificate verification error at depth 1: unable to get local issuer certificate 2024-06-04T15:15:36.191711347+02:00 [2024-06-04 13:15:36.191][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:241] [Tags: "ConnectionId":"8"] remote address:10.X.X.254:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

[optional Relevant Links:]

Any extra documentation required to understand the issue. https://gateway.envoyproxy.io/latest/tasks/security/jwt-authentication/ https://gateway.envoyproxy.io/contributions/design/security-policy/

[arkodg]

I'd vote for adding a caCertificateRefs field in here to solve this

[phantooom]

/assign

[phantooom]

I'd vote for adding a caCertificateRefs field in here to solve this

is caCertificateRef field ok？ is there a scenario where users might need multiple CAs?

[arkodg]

I'd vote for adding a caCertificateRefs field in here to solve this

is caCertificateRef field ok？ is there a scenario where users might need multiple CAs?

@phantooom lets follow the BackendTLSPolicy spec https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/ as a guide and use caCertificateRefs and append them into one internally

[phantooom]

I'd vote for adding a caCertificateRefs field in here to solve this

is caCertificateRef field ok？ is there a scenario where users might need multiple CAs?

@phantooom lets follow the BackendTLSPolicy spec https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/ as a guide and use caCertificateRefs and append them into one internally

ok

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

hey @phantooom this was discussed in the the community meeting a few weeks ago and the decision is to represent the remote jwks endpoint as a backendCluster. similar to what's done in ext auth https://github.com/envoyproxy/gateway/blob/262e0466f14dace834f1b0d712c2492c27e9bb03/api/v1alpha1/ext_auth_types.go#L54

This would be an additional field in the existing struct allowing the user to define custom cluster level properties like custom caCert

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[s0uky]

+1 Is there any progress on this feat?

[arkodg]

hey @phantooom do you any cycles to address the above comments ? this is similar to https://github.com/envoyproxy/gateway/pull/4227

[phantooom]

hey @phantooom do you any cycles to address the above comments ?嘿，你有空处理上面的评论吗？ this is similar to #4227这与 #4227 类似

sorry. my work has recently changed, and I am no longer able to continue following up on this pull request.

[phantooom]

/unassign

[wengyao04]

Hi, we also need remoteJWKS to support extra CA, do we have any process for this issue ?

[arkodg]

hey @wengyao04 to implement this, we'd need to add and implement the backendRefs field similar to what was done for OIDC https://github.com/envoyproxy/gateway/pull/4227, which would allow the user to specify a Backend/ Service resource as a backendRef + a BackendTLSPolicy as a policy to specify the custom CA

[MLNW]

As far as I understand the comments there is currently no way to use an internal JWKS server. Is that correct?

To give some context here is my situation: I have a local Kubernetes cluster with Keycloak, Envoy Gateway and a self-signing cert-manager instance. A backend service is exposed via the Envoy Gateway and I want to add JWT authentication based on JWTs issued by my Keycloak instance.

When I curl the endpoint with authentication enabled I get this error: Jwks remote fetch is failed. When using the verbose switch I see this in the response:

curl -kv --header "Authorization: Bearer $BEARER" https://api.dev.example.com/service/resource
...
* Server certificate:
*  subject: CN=Example
*  start date: Nov 21 14:14:43 2024 GMT
*  expire date: Feb 19 14:14:43 2025 GMT
*  issuer: CN=Example
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
...
< HTTP/2 401
< www-authenticate: Bearer realm="https://api.dev.example.com/service/resource", error="invalid_token"
...
Jwks remote fetch is failed

I would really like to move on with configuring authentication (and later authorization) in my development cluster but this is stopping me in my tracks. Is there a workaround or is this simply not supported until this issue is resolved?

[larssb]

Lucas yes there is. We do so in our prod. and test setup. Keycloak being the JWK broker and enovy the entrance gateway to authenticate through. I'm not at the work computer right now. But, if interested I can try to fetch some detail.s

On Sun, Dec 8, 2024 at 11:18 AM Lucas Resch @.***> wrote:

As far as I understand the comments there is currently no way to use an internal JWKS server. Is that correct?

To give some context here is my situation: I have a local Kubernetes cluster with Keycloak, Envoy Gateway and a self-signing cert-manager instance. A backend service is exposed via the Envoy Gateway and I want to add JWT authentication based on JWTs issued by my Keycloak instance.

When I curl the endpoint with authentication enabled I get this error: Jwks remote fetch is failed. When using the verbose switch I see this in the response:

curl -kv --header "Authorization: Bearer $BEARER" https://api.dev.example.com/service/resource ...

• Server certificate:
• subject: CN=Example
• start date: Nov 21 14:14:43 2024 GMT
• expire date: Feb 19 14:14:43 2025 GMT
• issuer: CN=Example
• SSL certificate verify result: self-signed certificate (18), continuing anyway.
• Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption ... < HTTP/2 401 < www-authenticate: Bearer realm="https://api.dev.example.com/service/resource", error="invalid_token" ... Jwks remote fetch is failed

I would really like to move on with configuring authentication (and later authorization) in my development cluster but this is stopping me in my tracks. Is there a workaround or is this simply not supported until this issue is resolved?

— Reply to this email directly, view it on GitHub https://github.com/envoyproxy/gateway/issues/3536#issuecomment-2525575748, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIERWXMSFMV57EGMP4ILRD2EQMGXAVCNFSM6AAAAABIY7UK5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRVGU3TKNZUHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Mvh. / Kind regards Lars Bengtsson

• Health is wealth, movement is medicine -

My DevOps blog https://bengtssondd.it My poetry website https://skakmatdigte.dk

[MLNW]

Lucas yes there is. We do so in our prod. and test setup. Keycloak being the JWK broker and enovy the entrance gateway to authenticate through. I'm not at the work computer right now. But, if interested I can try to fetch some detail.s

I'm certainly interested. Any kind of guidance is much appreciated! From the docs alone I can't seem to make it work.

[MLNW]

I was able to resolve my issue. First I enabled signing of my certificates via Let's Encrypt. Then I resolved the Jwks remote fetch is failed error by pointing the jwksURL to the cluster internal URL of my Keycloak instance instead of the URL exposed by the Envoy gateway. I guess that URL is not resolvable by Envoy itself.