Kiam Agents logging 403 errors as 500 errors

[awwithro]

When credentials are requested for a role that the kiam server is unable to assume, AWS sends a 403 error back to the server which is expected. However, the agent requesting these credentials logs the error as a 500. I think the 403 status code should be used since it improves monitoring of the kiam components to differentiate between authz errors and server errors.

Example:

{"addr":"192.168.208.57:42460","level":"error","method":"GET","msg":"error processing request: error fetching credentials: rpc error: code = Unknown desc = AccessDenied: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/kiam-server-role/1575495558438194904 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/some-other-role\n\tstatus code: 403, request id: 70f30c1b-2d88-11ea-b86c-c744558a598d","path":"/latest/meta-data/iam/security-credentials/arn:aws:iam::xxxxxxxxxxxx:role/some-other-role","status":500,"time":"2020-01-02T17:51:02Z"}

[awwithro]

The metric in question is: kiam_metadata_responses_total

[savvypearl]

i am currently having this issue, please how did you resolve this?