Internal Server Error when doing POST to /oauth2/token during Code Grant flow when exchanging Code on Token

[denysandriyanov]

Preflight checklist

• [X] I could not find a solution in the existing issues, docs, nor discussions.
• [X] I agree to follow this project's Code of Conduct.
• [X] I have read and am following this repository's Contribution Guidelines.
• [X] I have joined the Ory Community Slack.
• [X] I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

During Code Grant flow after code was obtained doing call to exchange code on tokens causes Internal Server error, please see details below:

Reproducing the bug

1. Run hydra
2. Execute code grant flow, obtain code from oaut2/auth
3. Do Post call to oauth2/token endpoint

Relevant log output

During Code Grant flow after code was obtained doing call to exchange code on tokens causes Internal Server error, please see details below:

{
  "audience": "application",
  "error": {
    "debug": "server_error",
    "message": "server_error",
    "reason": "",
    "stack_trace": "\ngithub.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).getKeys\n\t/project/jwk/jwt_strategy.go:54\ngithub.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).getPrivateKey\n\t/project/jwk/jwt_strategy.go:76\n... (truncated for brevity)",
    "status": "Internal Server Error",
    "status_code": 500
  },
  "http_request": {
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip,deflate",
      "authorization": ["Basic ZDMwNDVmZTAtZTlmNS00ZDM3LTg2NzQtMjYwZmIyNGVlMzhjOnNlY3JldA=="],
      "content-length": "169",
      "content-type": "application/x-www-form-urlencoded; charset=ISO-8859-1",
      "traceparent": "00-03a4a331ff249118b8e8826e695d4688-c98ac7f57b917007-01",
      "uber-trace-id": "03a4a331ff249118b8e8826e695d4688:c98ac7f57b917007:0:1",
      "user-agent": "Apache-HttpClient/4.5.14 (Java/17.0.9)",
      "x-b3-sampled": "1",
      "x-b3-spanid": "c98ac7f57b917007",
      "x-b3-traceid": "03a4a331ff249118b8e8826e695d4688",
      "x-forwarded-port": "443",
      "x-forwarded-scheme": "https",
      "x-real-ip": "172.31.239.26",
      "x-request-id": "0703c9f6f7f09c00f8cade6a15604e2d",
      "x-scheme": "https"
    },
    "host": "proxy-upstream8130.us-east-1.cloud-int.avid.com",
    "method": "POST",
    "path": "/oauth2/token",
    "query": null,
    "remote": "10.81.30.33:59492",
    "scheme": "http"
  },
  "level": "error",
  "msg": "An error occurred",
  "otel": {
    "span_id": "0a33ec1b2d87517b",
    "trace_id": "03a4a331ff249118b8e8826e695d4688"
  },
  "service_name": "Ory Hydra",
  "service_version": "v2.1.2",
  "time": "2023-12-22T12:56:18.804941895Z"
}



Please see Hydra stack trace:
github.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).getKeys
    /project/jwk/jwt_strategy.go:54
github.com/ory/hydra/v2/jwk.(*DefaultJWTSigner).getPrivateKey
    /project/jwk/jwt_strategy.go:76
github.com/ory/fosite/token/jwt.(*DefaultSigner).Generate
    /go/pkg/mod/github.com/ory/[email protected]/token/jwt/jwt.go:44
github.com/ory/fosite/handler/oauth2.(*DefaultJWTStrategy).generate
    /go/pkg/mod/github.com/ory/[email protected]/handler/oauth2/strategy_jwt.go:131
github.com/ory/fosite/handler/oauth2.(*DefaultJWTStrategy).GenerateAccessToken
    /go/pkg/mod/github.com/ory/[email protected]/handler/oauth2/strategy_jwt.go:42
github.com/ory/hydra/v2/fositex.TokenStrategy.GenerateAccessToken
    /project/fositex/token_strategy.go:44
github.com/ory/fosite/handler/oauth2.(*AuthorizeExplicitGrantHandler).PopulateTokenEndpointResponse
    /go/pkg/mod/github.com/ory/[email protected]/handler/oauth2/flow_authorize_code_token.go:142
github.com/ory/fosite.(*Fosite).NewAccessResponse
    /go/pkg/mod/github.com/ory/[email protected]/access_response_writer.go:24
github.com/ory/hydra/v2/oauth2.(*Handler).oauth2TokenExchange
    /project/oauth2/handler.go:928
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2122
github.com/ory/x/httprouterx.NoCacheHandler.func1
    /go/pkg/mod/github.com/ory/[email protected]/httprouterx/nocache.go:37
... (truncated for brevity)

Relevant configuration

No response

Version

v2.1.2

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

This occurs only under heavy load. If single request executed everything is okis

[alnr]

Hydra generates its own JWKS on-demand if it can't find an existing one. Because that's typically an RSA key, this operation is cpu intensive.

I'm sure this could be handled better in the code, but should not be a common issue in production deployments where keys are long lived.