[Bug] EKSCTL_ENABLE_CREDENTIAL_CACHE=1 does not work in 0.165.0

[koooge]

Hi there,

What were you trying to accomplish?

EKSCTL_ENABLE_CREDENTIAL_CACHE=1 does not work to cache MFA token in 0.165.0. It worked in 0.160.0.

What happened?

$ eksctl get cluster
Assume Role MFA token code: <code>
Assume Role MFA token code: <code>
Assume Role MFA token code: <code>
Error: failed to list clusters in region "<region>": operation error EKS: ListClusters, get identity: get credentials: failed to refresh cached credentials, failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: <id>, api error AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.

Actually it created the cache file. But the expiration is set at just that point.

$ cat ~/.eksctl/cache/credentials.yaml
profiles:
  <name>:
    credential:
      accesskeyid: <id>
      secretaccesskey: <secret>
      sessiontoken: <token>
      providername: AssumeRoleProvider
    expiration: 2023-12-11T14:07:04Z # <<--- Will expire in the moment

How to reproduce it?

$ eksctl version
0.165.0

$ rm ~/.eksctl/cache/credentials.yaml
$ export EKSCTL_ENABLE_CREDENTIAL_CACHE=1
$ eksctl get cluster
2023-12-11 15:04:23 [!]  cache file /Users/whoami/.eksctl/cache/credentials.yaml does not exist.
2023-12-11 15:04:23 [!]  cache file /Users/whoami/.eksctl/cache/credentials.yaml does not exist.
Assume Role MFA token code:

Logs

Anything else we need to know?

As a workaround, I manually edited the expiration in ~/.eksctl/cache/credentials.yaml to a future datetime and it worked.

Versions

$ eksctl info
eksctl version: 0.165.0
kubectl version: v1.28.4
OS: darwin

[github-actions[bot]]

Hello koooge :wave: Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[TiberiuGC]

Hi @koooge - in 0.162.0, eksctl introduced an expiry window for aws credentials of 30 mins - https://github.com/eksctl-io/eksctl/pull/7116 - the motivation behind this being that some CloudFormation operations take long to complete and we don't want temporary credentials to expire during such operations.

Given the error you're facing, I'm guessing your credentials would have expired in less than 30 mins from when you ran the command. Therefore, the sdk tries to refresh the credentials but can't do so using the MFA token as it's no longer valid?

[koooge]

Hi @TiberiuGC , Yeah. the created expiration is too soon.

$ date
Tue Dec 12 17:23:46 CET 2023
$ rm -f ~/.eksctl/cache/credentials.yaml
$ eksctl get cluster
2023-12-12 17:23:53 [!]  cache file /Users/whoammi/.eksctl/cache/credentials.yaml does not exist.
2023-12-12 17:23:53 [!]  cache file /Users/whoami/.eksctl/cache/credentials.yaml does not exist.
Assume Role MFA token code: <token>
$ cat ~/.eksctl/cache/credentials.yaml
profiles:
...
    expiration: 2023-12-12T16:24:04Z

[marcelocg]

I have a similar issue. I am already logged in and have a valid session token in ~/.aws/credentials. Everytime I use eksctl it complains about the missing ~/.eksctl/cache/credentials but works fine nonetheless without asking me for login data, MFA, nothing. And it doesn't create the credentials cache file either.

[colinnutrislice]

Same issue for me in 165/166. Falling back to 160 fixed it.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[koooge]

Let me remove the stale label.

[gaetansnl]

I can confirm this issue on 171, and for me even with the cache disabled MFA isn't working. The only way I was able to connect is by modifying the expiration

[vutny]

I can confirm the same issue with v0.172.0. It seems that expiration value in ~/.eksctl/cache/credentials.yaml file is always set to the current time, so credentials are being "expired" immediately.

Here in the code it is set to refresh the creds 30 mins before they expire: https://github.com/eksctl-io/eksctl/blob/main/pkg/eks/apiv2.go#L83 And here the session duration is also set to 30 mins: https://github.com/eksctl-io/eksctl/blob/main/pkg/eks/apiv2.go#L72 That means it will be always '0' minutes interval for temporary creds validity.

So either need to increase sessions duration to 1 hour, which is default for assumable role. Or decrease ExpiryWindow property to 15 minutes, having automatic token refresh within 15 minutes.

[yuxiang-zhang]

@vutny thanks for the explanation. I think increasing sessions duration makes sense, would you like to open a PR for it?

[vutny]

Sounds good, @yuxiang-zhang ! Submitted PR #7626

[koooge]

I verified EKSCTL_ENABLE_CREDENTIAL_CACHE worked in 0.173.0. Thank you @vutny !