aws
EKS kubeconfig profile setting doesn't work when using environment varialbles for authentication
Describe the bug
When leveraging and external authentication application (i.e. jumpcloud) to generate AWS credentials they are set as environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If referencing them in a profile to assume a role in another account and using the --profile cli option with aws eks update-kubeconfig it adds the profile to the token generation mechanism using the AWS_PROFILE environment variable.
https://github.com/aws/aws-cli/blob/e8110b30fd079020af12c468f67867ef9fc69d01/awscli/customizations/eks/update_kubeconfig.py#L336-L340
Based on the documentation https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#id1 this will get ignored on kubectl commands because the other environment variables are already set. If the command is manually changed in the kubeconfig file to include the profile with the --profile argument then authentication works successfully. Perhaps this conditional could be changed to extend the executable arguments just like are done for the role_arn in line 330?
Expected Behavior
aws --profile XXXX eks update-kubeconfig --region us-west-1 --name eks-cluster-name would update the kubeconfig file with a command that allows kubectl to be run successfully for that context.
Current Behavior
The generated kubeconfig does not work for that context. Manually editing the kubeconfig file and adding argments for --profile XXXXXX allow kubectl to run successfully.
Reproduction Steps
Set environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with valid credentials to and AWS user that can assume a role in another account. Then used those for authentication to the second AWS account via ~/.aws/config with
[profile account2]
region = us-east-1
role_arn = arn:aws:iam::XXXXXXXXXXXX:role/AssumeRole
credential_source = Environment
Create an EKS cluster in the second account with the assumed role.
Generate a kubeconfig with aws --profile account2 eks update-kubeconfig --region {region} --name {eks-cluster-name}. Try to use that kubeconfig to interact with the cluster.
Possible Solution
if self._session.profile:
generated_user["user"]["exec"]["args"].extend([
"--profile",
self._session.profile)
])
Additional Information/Context
No response
CLI version used
aws-cli/2.11.6 Python/3.11.2
Environment details (OS name and version, etc.)
Darwin/22.3.0
Hi @tsaxon13 - thanks for reaching out. I attempted to reproduce the similar behavior with the steps provided and unfortunately, I was't able to. I was able to run the command without --profile. Did you run into any errors when you run it? Could you perhaps share your debug as it would give us more insight to finding the root cause? You could retrieve the logs by adding --debug to your command.
Best,
John
@aBurmeseDev Thanks for looking at it! I'm not sure if I quite explained it right. Perhaps this will explain it a little better.
Here is a full example of what I'm doing and the error I get (with sensitive info redacted):
# Generate my AWS Auth information
➜ ~ aws-jumpcloud rotate bcic_prod
Using JumpCloud login details from your OS keychain.
Enter your JumpCloud multi-factor auth code: XXXXXX
Temporary IAM session for "bcic_prod" removed.
Attempting SSO authentication to Amazon Web Services...
AWS temporary session rotated; new session valid until Wed Apr 5 23:12:34 2023 UTC.
# Add it to my environment
➜ ~ source <(aws-jumpcloud export bcic_prod)
➜ ~ env | grep AWS
AWS_ACCESS_KEY_ID=REDACTED
AWS_SECRET_ACCESS_KEY=REDACTED
AWS_SECURITY_TOKEN=REDACTED
AWS_SESSION_TOKEN=REDACTED
# Generate a kubeconfig file from a cluster in a profile that I'm assuming a role in
➜ ~ aws --profile rlb-int eks update-kubeconfig --kubeconfig ~/kubeconfig-test --name ib-dev-c1
Updated context arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1 in /Users/anthonysaxon/kubeconfig-test
➜ ~ cat ~/kubeconfig-test
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: REDACTED
name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
contexts:
- context:
cluster: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
user: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
current-context: arn:aws:eks:us-east-1:REDATCTED:cluster/ib-dev-c1
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- ib-dev-c1
- --output
- json
command: aws
env:
- name: AWS_PROFILE
value: rlb-int
# Using that config results in unauthorized because it is using the auth information for my master account from my environment variables not the rlb-int profile
➜ ~ kubectl --kubeconfig ~/kubeconfig-test get nodes
error: You must be logged in to the server (Unauthorized)
If I manually edit the kubeconfig-test file and add --profile as an argument to the token generation command, everything works.
➜ ~ vi kubeconfig-test
➜ ~ cat ~/kubeconfig-test
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: REDACTED
name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
contexts:
- context:
cluster: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
user: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
current-context: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:REDACTED:cluster/ib-dev-c1
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- us-east-1
- --profile
- rlb-int
- eks
- get-token
- --cluster-name
- ib-dev-c1
- --output
- json
command: aws
env:
- name: AWS_PROFILE
value: rlb-int
➜ ~ kubectl --kubeconfig ~/kubeconfig-test get nodes
NAME STATUS ROLES AGE VERSION
ip-10-237-106-65.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-110-150.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-118-187.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-124-213.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-73-41.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-76-120.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-83-123.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
ip-10-237-93-112.ec2.internal Ready <none> 37d v1.24.10-eks-48e63af
For reference my ~/.aws/config looks like this
[profile bcic_prod]
region = us-east-1
role_arn = arn:aws:iam::ACCOUNT1REDACTED:role/SAML_ADMIN
credential_source = Environment
[profile rlb-int]
region = us-east-1
role_arn = arn:aws:iam::ACCOUNT2REDACTED:role/BCICProd-AssumeRole
credential_source = Environment
So my aws cli commands are not what is giving me the error, it's just generating a kubeconfig file that is not compatible with the authentication configuration that we need to use. If the token generation command that is created in the kubeconfig file used the --profile argument instead of the ENV: section, it appears that this would not happen. Let me know if that makes more sense or if more info is needed.
i think i'm hitting the same bug. i fixed it by manually editing the kubeconfig from:
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- my-cluster
command: aws
env:
...
- name: AWS_PROFILE
value: my-profile
to
...
env:
...
- name: AWS_PROFILE
value: my-profile
- name: AWS_SECRET_ACCESS_KEY
value: ""
- name: AWS_ACCESS_KEY_ID
value: ""
- name: AWS_SESSION_TOKEN
value: ""
which fixed the problem for me.
(seems like it would be straightforward to change the lines in update-kubeconfig.py to set those env variables to empty in the same place where it sets AWS_PROFILE... happy to send a PR for that if a maintainer would be open to accepting that change)
I faced the same issue in the GitLab pipeline
Thank you @tsaxon13 for the link to the documentation
In my case, it was possible to workaround the issue by unsetting AWS variables before calling kubectl
debug:
image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:v0.3.11
stage: test
variables:
KUBECTL_VERSION: "v1.21.14"
script:
- mkdir -p ~/.aws
- cp $aws_config ~/.aws/config
- curl -LO https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl
- install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
- aws --profile production eks update-kubeconfig --region eu-central-1 --name prod
- unset AWS_ACCESS_KEY_ID
- unset AWS_SECRET_ACCESS_KEY
- kubectl get po
GitLab variable aws_config
Type: file
Contents:
[profile default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
[profile production]
role_arn = arn:aws:iam::111222333:role/OrganizationAccountAccessRole
source_profile = default
+1 for the proposed by @tsaxon13 solution. I do not understand the reason why aws cli adds AWS_PROFILE instead of --profile in .kube/config
Running into this issue as well. We're using SSO and cross-account roles to access our EKS clusters, so it's frustrating when we have credentials from one account being passed to a different account incorrectly.
Can confirm both @tsaxon13's and @nicks's manual config modifications work for me. Our guidance to our devs has been to tweak their config with yq for an automated version of @tsaxon13's solution:
CLUSTER_NAME=cluster-1
CLUSTER_PROFILE=cluster-profile
aws eks update-kubeconfig --name $CLUSTER_NAME --region us-east-1 --profile $CLUSTER_PROFILE
yq -i "(.users[] | select(.name == \"$CLUSTER_NAME\") | .user.exec.args) += [\"--profile\", \"$CLUSTER_PROFILE\"]" ~/.kube/config
To make this more likely to be fixed, would it be preferable to have the behavior reflect how the profile is passed to update-kubeconfig? For example:
aws eks update-kubeconfig --profile profile-namewill append args with--profile,profile-nameaws eks update-kubeconfigwithAWS_PROFILEset will append env withAWS_PROFILEaws eks update-kubeconfigwith noAWS_PROFILEset will not append env
confirming that @calebmckay workaround worked for us too with minor change:
CLUSTER_NAME=cluster-1
CLUSTER_PROFILE=cluster-profile
aws eks update-kubeconfig --name $CLUSTER_NAME --region us-east-1 --profile $CLUSTER_PROFILE
yq -i "(.users[] | select(.name | contains( \"$CLUSTER_NAME\")) | .user.exec.args) += [\"--profile\", \"$CLUSTER_PROFILE\"]" ~/.kube/config
diff:
CLUSTER_NAME=cluster-1
CLUSTER_PROFILE=cluster-profile
aws eks update-kubeconfig --name $CLUSTER_NAME --region us-east-1 --profile $CLUSTER_PROFILE
-- yq -i "(.users[] | select(.name == \"$CLUSTER_NAME\") | .user.exec.args) += [\"--profile\", \"$CLUSTER_PROFILE\"]" ~/.kube/config
++ yq -i "(.users[] | select(.name | contains( \"$CLUSTER_NAME\")) | .user.exec.args) += [\"--profile\", \"$CLUSTER_PROFILE\"]" ~/.kube/config
And I agree with his suggestions.
This issue is broader than just cases where credentials are set as environment variables (i.e. AWS_ACCESS_KEY). If the profile passed as an environment variable uses a credentials provider that is further back in the resolver chain, the profile will not be used.
For example, if we are running in a Kubernetes Pod and have IRSA (i.e the AssumeRoleWithWebIdentity provider), it's impossible to use a profile that leverages the Process provider because the default ordering puts WebIdentity earlier. The default resolution order is:
- EnvProvider
- AssumeRoleProvider
- AssumeRoleWithWebIdentityProvider
- SSOProvider
- SharedCredentialProvider
- ProcessProvider
