Very high CPU usage when MFA expires when running --ec2-server

[pjroth]

When my MFA session expires I sometimes get the following behavior where I exit the server (it is usually running from my previous work day) and then launch it again. I am prompted for MFA at this time (notice below I entered my token). Then you can see in below output that soon after the debug output indicates the credentials expired and there was a prompt for MFA. The problem is I'm usually no longer looking at this terminal output so it might sit there for awhile. When this happens my CPU usage is very high. Eventually I notice my fan is running and using htop or similar I notice aws-vault is using 300+ percent of my 4 CPUs!

If I exit the aws-vault process and launch it again (entering my MFA code when prompted) then CPU returns back to "normal" levels.

• [ ] I am using version v6.2.0 release of AWS Vault but I'll upgrade to latest and report back if this occurs with that version as well)
• [ ] my .aws/config withonly the default profile (redacted)

  [default]
  output=json
  region=us-east-1
  mfa_serial=arn:aws:iam::1234567890:mfa/myusername
  credential_process=aws-vault exec default --json --prompt=terminal --duration=12h
  

• [ ] I have provided the debug output using aws-vault --debug (redacted)

# aws-vault exec default --debug --prompt=terminal --duration=20h --ec2-server
2021/05/18 08:25:33 aws-vault v6.2.0
2021/05/18 08:25:33 Loading config file /home/myusername/.aws/config
2021/05/18 08:25:33 Parsing config file /home/myusername/.aws/config
2021/05/18 08:25:33 [keyring] Considering backends: [secret-service]
2021/05/18 08:25:33 profile default: using stored credentials
2021/05/18 08:25:33 profile default: using GetSessionToken (with MFA)
Enter token for arn:aws:iam::1234567890:mfa/myusername: 123456
2021/05/18 08:25:47 Using STS endpoint https://sts.amazonaws.com
2021/05/18 08:25:47 Looking up keyring for 'default'
2021/05/18 08:25:47 Setting subprocess env: AWS_DEFAULT_REGION=us-east-1, AWS_REGION=us-east-1
2021/05/18 08:25:47 Starting child process: /usr/bin/zsh 
2021/05/18 08:25:47 Starting EC2 Instance Metadata server on 127.0.0.1:9099
2021/05/18 08:26:26 http: 127.0.0.1:33552: 404 PUT /latest/api/token (7.769µs)
2021/05/18 08:26:26 http: 127.0.0.1:33552: 404 PUT /latest/api/token (27.28µs)
2021/05/18 08:26:26 http: 127.0.0.1:33552: 404 PUT /latest/api/token (22.154µs)
2021/05/18 08:26:26 http: 127.0.0.1:33552: 200 GET /latest/meta-data/iam/security-credentials/ (17.907µs)
2021/05/18 08:26:26 Credentials.IsExpired() = true
Enter token for arn:aws:iam::1234567890:mfa/myusername: 2021/05/18 08:26:27 Credentials.IsExpired() = true
2021/05/18 08:26:28 Credentials.IsExpired() = true
2021/05/18 08:26:29 http: 127.0.0.1:33564: 200 GET /latest/meta-data/iam/security-credentials/ (4.522µs)
2021/05/18 08:26:29 Credentials.IsExpired() = true
2021/05/18 08:26:30 Credentials.IsExpired() = true
2021/05/18 08:26:31 Credentials.IsExpired() = true
2021/05/18 08:26:32 http: 127.0.0.1:33576: 404 PUT /latest/api/token (6.317µs)
2021/05/18 08:26:32 http: 127.0.0.1:33576: 404 PUT /latest/api/token (6.844µs)
2021/05/18 08:26:32 http: 127.0.0.1:33576: 404 PUT /latest/api/token (7.562µs)
2021/05/18 08:26:32 http: 127.0.0.1:33576: 200 GET /latest/meta-data/iam/security-credentials/ (4.728µs)
2021/05/18 08:26:32 Credentials.IsExpired() = true

[mtibben]

If you're using credential_process=aws-vault you shouldn't be executing aws-vault via the CLI else you'll have a infinite loop. This will be the cause of your 100% CPU load

[pjroth]

@mtibben that is interesting... I've never ran into this before though. When I launch the via --ec2-server for example it prompts for my MFA, I enter, and it works just fine. It's only when my MFA session expires that this happens. Maybe it's because the prompt is sitting there for a long time (until I notice which could be hours if I don't need AWS access for awhile). It seems like it should prompt me the same way it prompted me originally when I ran the command and not examine the credential_process config? Perhaps there is a technical hurdle to implement this way I'm not sure. Or, perhaps the prompt always consumes a lot of CPU and normally I enter it quickly so I don't notice it.

Having the credential_process in my config allows me to both use the metadata server via --ec2-server (my most common use case) and the credential_process for when I forget to launch the metadata server. Since it's manual right now (I run the command each morning in a long running shell) for me to launch this so sometimes I forget. Perhaps it would be nice if there was a way to automate this launch so I don't have to rely on the "fallback" of the credential_process? Granted that is a different feature.

Thank you!

[pjroth]

Another interesting thing to note. I get prompted to enter creds in the output but there is no way to enter them. I believe because after the first successful execution of the command I get my prompt back (I believe I am dropped into a new shell). Then the prompt happens after my MFA session expires but there is no way to enter this MFA input.

I would actually prefer that when I run --ec2-server it runs like a standard daemon process and doesn't give my prompt/shell back since I don't need/want that shell when I'm running the metadata server. It actually causes more confusion since I now need to leave a shell open that "looks" normal but is actually going to exit the metadata server if I type exit. That might then also allow for the MFA prompt to work properly if there is no new shell.

Thanks for this great tool!

[ulrichwinter]

I experienced the same behaviour after using "aws-vault exec some-profile --server" locally on macOS. The cpu usage was very high (activity monitor reported 230%). The terminal, where the above command had been invoked, showed a prompt to enter the MFA code (which I did not try).

I'm using aws-vault: 6.3.1 installed via homebrew. The MFA setting is: credential_process=aws-vault exec profilename --json

[mtibben]

yeah - again you can't use aws-vault CLI when using credential_process=aws-vault

[ulrichwinter]

Thanks Michael. But: Today I removed that credential_process setting from my aws config and got the same problem.

Here the config:

[default]
region=eu-central-1

[profile baseprofile]
mfa_serial=arn:aws:iam::123456789012:mfa/MyMFA
source_profile=baseprofile

[profile used-profile]
include_profile=baseprofile
role_arn=arn:aws:iam::987654321098:role/some-role

Command used: aws-vault exec used-profile --debug --server

Now, after several hours, some other terminal where I use k9s, prints many times the following message: `Unable to locate credentials. You can configure credentials by running "aws configure".

Am I still doing something wrong?

My intention in using that Local Instance Metadata method of passing credentials is to use shorter session timeouts. As stated in the "How it works" section of the aws-vault Readme. In fact, I want to avoid having to provide the MFA code more often than once when invoking the aws-vault server on my local machine.

Is this intention anyhow feasible?

[pavelbrylov]

I am not sure, but it sounds like the issue is because once MFA expires, aws-vault is waiting for input endlessly in a detached terminal? Me and my colleagues had the same issue, and solution was pretty simple - to use GUI for MFA request: just add --prompt=osascript to your --server calls or set env var AWS_VAULT_PROMPT=osascript. Turns out it is all there in the docs! :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.