How do I tell if --assume-role-arn is successful?

[d-shi]

Hi, I have followed all the docs, and have the server and agent daemonsets up and running. I have configured the server role, node role for nodes where the server runs, and pod role, as well as applied the pod and namespace annotations.

I am now seeing this access denied error in the server logs:

{"level":"error","msg":"error requesting credentials: AccessDenied: Access denied\n\tstatus code: 403, request id: <redacted>","pod.iam.role":"<redacted>","time":"2019-07-24T23:17:09Z"}

I have supplied the ARN of the server role via the --assume-role-arn flag, but I do not know how to tell if the server is successfully assuming this role. There is a log message for detecting arn prefix but not for assuming the role. I tried supplying a fake non-existent ARN and nothing changed in the logs. However, if I remove the trailing doublequote after the ARN, then I get this message:

error requesting credentials: ValidationError: Requested ARN is invalid.\n\tstatus code: 400

It feels like somehow the server is not successfully assuming the provided role, but it is hard to verify. Any help would be appreciated. Thanks!

[d-shi]

Some more info:

If I grant assume all roles permission (normally granted to the server role) to the server node role, change the trust relationship of my pod roles to the server node role, and no longer specify --assume-role-arn in the kiam-server args, then everything works.

It seems to me something is either misconfigured on my end which I cannot find, or something is wonky with the --assume-role-arn flag. I have tested manually assuming the server role and then assuming a pod role manually from the server node, which works.