Oathkeeper returns one of two messages for the same error

[DeppLearning]

Preflight checklist

• [X] I could not find a solution in the existing issues, docs, nor discussions.
• [X] I agree to follow this project's Code of Conduct.
• [X] I have read and am following this repository's Contribution Guidelines.
• [ ] This issue affects my Ory Cloud project.
• [ ] I have joined the Ory Community Slack.
• [ ] I am signed up to the Ory Security Patch Newsletter.

Describe the bug

I am sometimes getting a different errors message for the same request. The route is protected by oathkeeper and the request I am sending does not contain any authentication.

Oathkeeper is correctly blocking the request, however, on rare occasions I am getting a different error message. Usually it's "Access credentials are invalid" but sometimes it's "The request could not be authorized". I'd expect to always see the same error message.

Reproducing the bug

I can't reliably reproduce the behavior.

Relevant log output

The odd response looks like this:

oathkeeper | {"http_request":{"headers":{"accept":"*/*","user-agent":"curl/7.79.1"},"host":"127.0.0.1:4455","method":"GET","path":"/api/v1/devices","query":null,"remote":"172.18.0.1:58236","scheme":"http"},"level":"info","msg":"started handling request","time":"2021-10-28T14:19:16Z"}
oathkeeper | {"audience":"application","error":{"debug":"","message":"Access credentials are invalid","reason":"","status":"Unauthorized","status_code":401},"granted":false,"http_host":"127.0.0.1:4455","http_method":"GET","http_url":"http://127.0.0.1:4455/api/v1/devices","http_user_agent":"curl/7.79.1","level":"warning","msg":"No authentication handler was responsible for handling the authentication request","reason_id":"authentication_handler_no_match","rule_id":"ig:backend:protected","service_name":"ORY Oathkeeper","service_version":"v0.38.15-beta.1","time":"2021-10-28T14:19:16Z"}
oathkeeper | {"audience":"application","error":{"debug":"","message":"Access credentials are invalid","reason":"","status":"Unauthorized","status_code":401},"granted":false,"http_host":"127.0.0.1:4455","http_method":"GET","http_url":"http://127.0.0.1:4455/api/v1/devices","http_user_agent":"curl/7.79.1","level":"warning","msg":"Access request denied","service_name":"ORY Oathkeeper","service_version":"v0.38.15-beta.1","time":"2021-10-28T14:19:16Z"}
oathkeeper | {"code":401,"debug":"","details":{},"error":"The request could not be authorized","level":"error","msg":"An error occurred while handling a request","reason":"","request-id":"","status":401,"time":"2021-10-28T14:19:16Z","writer":"JSON"}
oathkeeper | {"http_request":{"headers":{"accept":"*/*","user-agent":"curl/7.79.1"},"host":"127.0.0.1:4455","method":"GET","path":"/api/v1/devices","query":null,"remote":"172.18.0.1:58236","scheme":"http"},"http_response":{"status":401,"text_status":"Unauthorized","took":1904787},"level":"info","msg":"completed handling request","time":"2021-10-28T14:19:16Z"}

Usually the response looks like this, note the different error message in the 4. log line.

oathkeeper | {"http_request":{"headers":{"accept":"*/*","user-agent":"curl/7.75.0"},"host":"127.0.0.1:4455","method":"GET","path":"/api/v1/devices","query":null,"remote":"172.18.0.1:62542","scheme":"http"},"level":"info","msg":"started handling request","time":"2021-10-28T14:37:41Z"}
oathkeeper | {"audience":"application","error":{"debug":"","message":"Access credentials are invalid","reason":"","status":"Unauthorized","status_code":401},"granted":false,"http_host":"127.0.0.1:4455","http_method":"GET","http_url":"http://127.0.0.1:4455/api/v1/devices","http_user_agent":"curl/7.75.0","level":"warning","msg":"No authentication handler was responsible for handling the authentication request","reason_id":"authentication_handler_no_match","rule_id":"ig:backend:protected","service_name":"ORY Oathkeeper","service_version":"v0.38.15-beta.1","time":"2021-10-28T14:37:41Z"}
oathkeeper | {"audience":"application","error":{"debug":"","message":"Access credentials are invalid","reason":"","status":"Unauthorized","status_code":401},"granted":false,"http_host":"127.0.0.1:4455","http_method":"GET","http_url":"http://127.0.0.1:4455/api/v1/devices","http_user_agent":"curl/7.75.0","level":"warning","msg":"Access request denied","service_name":"ORY Oathkeeper","service_version":"v0.38.15-beta.1","time":"2021-10-28T14:37:41Z"}
oathkeeper | {"code":401,"debug":"","details":{},"error":"Access credentials are invalid","level":"error","msg":"An error occurred while handling a request","reason":"","request-id":"","status":401,"time":"2021-10-28T14:37:41Z","writer":"JSON"}
oathkeeper | {"http_request":{"headers":{"accept":"*/*","user-agent":"curl/7.75.0"},"host":"127.0.0.1:4455","method":"GET","path":"/api/v1/devices","query":null,"remote":"172.18.0.1:62542","scheme":"http"},"http_response":{"status":401,"text_status":"Unauthorized","took":44441600},"level":"info","msg":"completed handling request","time":"2021-10-28T14:37:41Z"}

Relevant configuration

I am using the oryd/oathkeeper:v0.38-alpine and oryd/kratos:v0.7.6-alpha.1 docker images (pretty sure kratos is not involved though)

I also set these in oathkeepers env vars:

ERRORS_HANDLERS_JSON_ENABLED=true
ERRORS_HANDLERS_JSON_CONFIG_VERBOSE=true
ERRORS_FALLBACK=redirect,json

and have this in my access rules for the route that is being hit:

authenticators: 
- handler: cookie_session
- handler: bearer_token
authorizer: 
handler: allow
mutators: 
- handler: noop
errors: 
- handler: json
  config:
    when:
      - error:
          - unauthorized
          - forbidden

Version

v0.38-alpine

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Docker Compose

Additional Context

After starting oathkeeper, I always get the same response, it's not that I have ever seen the same instance of oathkeeper return either message.

For me that itself is not a serious issue, I'm just worried about some misconfiguration on my side or this being a symptome of some more serious issue.

[DeppLearning]

This is probably a very weird bug in one of my dependencies, sorry for the noise, will reopen if the issue appears again.

edit: Issue persists in different environments

[kmherrmann]

Please re-open this if problems persist.