amazon-ecr-credential-helper
amazon-ecr-credential-helper copied to clipboard
awslabs
Unable to Use amazon-ecr-credential-helper with OIDC Token
Issue Description:
I am encountering an issue while attempting to use the amazon-ecr-credential-helper in combination with an OIDC token for authentication. The goal is to push a Docker image to an Amazon ECR registry using Kaniko within a specific context. However, the process is failing with authentication errors.
Steps to Reproduce:
-
Create the necessary directory structure and files:
$ mkdir -p /kaniko/.aws $ echo "${MY_OIDC_TOKEN}" > /kaniko/web_identity_token $ echo -e "[default]\nrole_arn=${AWS_ROLE_ARN}\nweb_identity_token_file=/kaniko/web_identity_token" > /kaniko/.aws/config $ mkdir -p /kaniko/.docker $ echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json -
Execute Kaniko:
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${ECR_REGISTRY}:${DOCKER_IMAGE_TAG}"
Expected Behavior:
I expected the Kaniko process to authenticate successfully using the amazon-ecr-credential-helper with the provided OIDC token, and for the Docker image to be pushed to the specified Amazon ECR registry.
Actual Behavior: The Kaniko process is failing with the following error messages:
SDK 2023/08/08 14:47:55 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 405, request to EC2 IMDS failed
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "000000000.dkr.ecr.us-east-1.amazonaws.com/test:kaniko-test-455a9c73": POST https://000000000.dkr.ecr.us-east-1.amazonaws.com/v2/test/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized
Additional Information:
- AWS IAM
{
"Action": [
"ecr:UploadLayerPart",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:GetDownloadUrlForLayer",
"ecr:CompleteLayerUpload",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "ECR"
},
{
"Action": "ecr:GetAuthorizationToken",
"Effect": "Allow",
"Resource": "*",
"Sid": "ECRGetAuthorizationToken"
}
- The provided OIDC token and Amazon ECR registry details are correct.
- The
amazon-ecr-credential-helperis expected to handle the authentication for ECR registry access. - The error messages suggest that there might be an issue with the authentication process or IAM permissions.
- The use of IMDSv1 is also causing warnings, which might be related to the authentication failure.
Environment:
- Kaniko Version: gcr.io/kaniko-project/executor:debug
Desired Solution:
I would appreciate assistance in resolving this issue. Specifically, I am looking for guidance on how to properly configure and use the amazon-ecr-credential-helper with an OIDC token to authenticate the Kaniko process for pushing Docker images to an Amazon ECR registry.
Thank you for your help!
(Just a drive-by observer, but) presumably you'd need to do an sts.assumeRoleWithWebIdentity i.e.:
aws sts assume-role-with-web-identity --role-arn <> --role-session-name <> --web-identity-token <>
first then then cred helper would work? And you'd need to set up your IAM role accordingly
Hey @pauldthomson, thank you for your response! Unfortunately, the image I'm using (kaniko) doesn't provide the option to install awscli. You could create a dedicated imag:
FROM alpine
RUN apk add --no-cache jq curl python3 py3-pip gettext libintl bash && pip install awscli
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/executor /kaniko/executor
But I was looking for that functionality in amazon-ecr-credential-helper, not in awscli. Otherwise, that should be a viable workaround.
Of course, I didn't think of the context of where Kaniko runs 🤦♂️
Are you running it in EKS? That would at least open the door to IRSA
You can use wget to call assume web identity. Although I found it was broken so patched it with an older version. You can use the example below in your gitlab jobs by using extend: .aws_login. This then exposes credentials via the standard ENV variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
.aws_login:
id_tokens:
MY_OIDC_TOKEN:
aud: https://gitlab.com
before_script:
- |
echo "Script here"
The script to just get kaniko working...
#!/bin/sh
if [ -n "$ROLE_ARN" ]; then
echo "Logging in with $ROLE_ARN"
else
echo "Please set ROLE_ARN in CICD variables"
exit 1
fi
# Regional endpoints eg. "https://sts.ap-southeast-2.amazonaws.com/"
sts_endpoint="https://sts.amazonaws.com/"
session_name="GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
duration_seconds="3600"
action="Action=AssumeRoleWithWebIdentity"
duration="DurationSeconds=${duration_seconds}"
role_arn="RoleArn=${ROLE_ARN}"
role_session="RoleSessionName=${session_name}"
web_identity="WebIdentityToken=${MY_OIDC_TOKEN}"
version="Version=2011-06-15"
params="${action}&${duration}&${role_arn}&${role_session}&${web_identity}&${version}"
content_type="Content-Type: application/x-www-form-urlencoded"
# Known bug https://github.com/GoogleContainerTools/kaniko/pull/2765
# Download version of wget that works with sts endpoint
wget -q --no-check-certificate https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox_WGET -O busybox_WGET
mv busybox_WGET /busybox/wget
chmod +x /busybox/wget
response=$(wget -q --no-check-certificate --header="$content_type" --post-data="$params" -O - "$sts_endpoint")
AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN
Alternatively, if you want multi container support, you could check for aws-cli, curl then wget.
#!/bin/sh
if [ -n "$ROLE_ARN" ]; then
echo "Logging in with $ROLE_ARN"
else
echo "Please set ROLE_ARN in CICD variables"
exit 1
fi
# Regional endpoints eg. "https://sts.ap-southeast-2.amazonaws.com/"
sts_endpoint="https://sts.amazonaws.com/"
session_name="GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
duration_seconds="3600"
action="Action=AssumeRoleWithWebIdentity"
duration="DurationSeconds=${duration_seconds}"
role_arn="RoleArn=${ROLE_ARN}"
role_session="RoleSessionName=${session_name}"
web_identity="WebIdentityToken=${MY_OIDC_TOKEN}"
version="Version=2011-06-15"
params="${action}&${duration}&${role_arn}&${role_session}&${web_identity}&${version}"
content_type="Content-Type: application/x-www-form-urlencoded"
if
command -v aws >/dev/null 2>&1
then
response=$(
aws sts assume-role-with-web-identity \
--role-arn "${ROLE_ARN}" \
--role-session-name "${session_name}" \
--web-identity-token "${MY_OIDC_TOKEN}" \
--duration-seconds "${duration_seconds}" \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text
)
# Set positional parameters
# shellcheck disable=SC2086
set -- $response
# Assign the credentials to environment variables
AWS_ACCESS_KEY_ID=$1
AWS_SECRET_ACCESS_KEY=$2
AWS_SESSION_TOKEN=$3
elif
command -v curl >/dev/null 2>&1
then
response=$(curl -s -X POST "$sts_endpoint" -H "$content_type" -d "$params")
AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')
elif
command -v wget >/dev/null 2>&1
then
wget_path=$(which wget)
if [ "$wget_path" = "/busybox/wget" ]; then
# Known bug https://github.com/GoogleContainerTools/kaniko/pull/2765
wget -q --no-check-certificate https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox_WGET -O busybox_WGET
mv busybox_WGET /busybox/wget
chmod +x /busybox/wget
response=$(wget -q --no-check-certificate --header="$content_type" --post-data="$params" -O - "$sts_endpoint")
else
response=$(
wget --quiet --method=POST --header="$content_type" --body-data="$params" --output-document - "$sts_endpoint"
)
fi
AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')
echo "AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID"
else
echo "Neither AWS CLI, curl, nor wget is available to call STS assume role.
Ensure your pipeline image contains one of these binaries."
exit 1
fi
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN
I have solved this in GitLab using environment variables. The credential helper is able to automatically assume a role given an OIDC token and a role ARN.
Example job in GitLab:
deploy_to_ecr:
stage: deploy
image:
name: gcr.io/kaniko-project/executor:v1.16.0-debug
entrypoint: [""]
id_tokens:
GITLAB_OIDC_TOKEN:
aud: https://gitlab.com
variables:
AWS_WEB_IDENTITY_TOKEN_FILE: /kaniko/gitlab-oidc-token
AWS_EC2_METADATA_DISABLED: "true"
AWS_SDK_LOAD_CONFIG: "true"
before_script:
- mkdir -p /kaniko/.docker
- echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
- echo $GITLAB_OIDC_TOKEN > $AWS_WEB_IDENTITY_TOKEN_FILE
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--target "lambda-ctx"
--destination "${ECR_REPO_URI}:${CI_COMMIT_SHORT_SHA}"
--skip-unused-stages=true
--snapshot-mode=redo
after_script:
- cat ~/.ecr/log/ecr-login.log
The important part of this setup involves saving the value of the GITLAB_OIDC_TOKEN environment variable to the path that is defined in the AWS_WEB_IDENTITY_TOKEN_FILE environment variable. Additionally, you need to set the AWS_ROLE_ARN environment variable somewhere (I used a variable via the CI/CD Settings for that one).
I would be curious to know if this works for anyone else, so please respond one way or another. The hoops that a lot of folks seem to be jumping through to make this work seem arduous, and hopefully with this solution it will be cleaner.
I have solved this in GitLab using environment variables. The credential helper is able to automatically assume a role given an OIDC token and a role ARN.
Example job in GitLab:
deploy_to_ecr: stage: deploy image: name: gcr.io/kaniko-project/executor:v1.16.0-debug entrypoint: [""] id_tokens: GITLAB_OIDC_TOKEN: aud: https://gitlab.com variables: AWS_WEB_IDENTITY_TOKEN_FILE: /kaniko/gitlab-oidc-token AWS_EC2_METADATA_DISABLED: "true" AWS_SDK_LOAD_CONFIG: "true" before_script: - mkdir -p /kaniko/.docker - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json - echo $GITLAB_OIDC_TOKEN > $AWS_WEB_IDENTITY_TOKEN_FILE script: - /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --target "lambda-ctx" --destination "${ECR_REPO_URI}:${CI_COMMIT_SHORT_SHA}" --skip-unused-stages=true --snapshot-mode=redo after_script: - cat ~/.ecr/log/ecr-login.logThe important part of this setup involves saving the value of the
GITLAB_OIDC_TOKENenvironment variable to the path that is defined in theAWS_WEB_IDENTITY_TOKEN_FILEenvironment variable. Additionally, you need to set theAWS_ROLE_ARNenvironment variable somewhere (I used a variable via the CI/CD Settings for that one).I would be curious to know if this works for anyone else, so please respond one way or another. The hoops that a lot of folks seem to be jumping through to make this work seem arduous, and hopefully with this solution it will be cleaner.
After busting my head for a whole day following the docs, using this EXACT configuration (same variable names, same paths) solved it for me.
Thanks for sharing! <3 !
After busting my head for a whole day following the docs, using this EXACT configuration (same variable names, same paths) solved it for me.
Thanks for sharing! <3 !
@DaniWS I'm so glad it worked for you! Happy to help ❤️