Having trouble with a shared GitLab runner and config files

[troyswanson]

Howdy, all -- I'm trying to build and push an image using Kaniko in a shared GitLab runner cluster. The IAM role for the instance that runs the job is not the one that I would like to use for the credential helper. I have credentials for an AWS user in our core users account which is meant to assume a role in our build account that has access to push to ECR.

---

I've configured the GitLab CI/CD variables to the following:

AWS_CONFIG (file)

[default]
region=us-east-1

[profile gitlab]
role_arn=arn:aws:iam::[account_id]:role/ServiceRoleForGitLabPipeline
source_profile=default

AWS_CREDENTIALS (file)

[default]
aws_access_key_id=[access_key_id]
aws_secret_access_key=[secret_access_key]

AWS_PROFILE (variable)

gitlab

ECR_REPO_URI (variable)

[account_id].dkr.ecr.us-east-1.amazonaws.com/lambda-wbxf

---

As a test, I've created a job that should output the session information of the assumed role:

get_credentials:
  stage: build
  image:
    name: amazon/aws-cli
    entrypoint: [""]
  before_script:
    - mkdir -p ~/.aws
    - cat $AWS_CONFIG > ~/.aws/config
    - cat $AWS_CREDENTIALS > ~/.aws/credentials
  script:
    - aws sts get-caller-identity

The output of this is exactly what I expect:

{
    "UserId": "AROA5QBFXVXRQOWNGV2A6:botocore-session-1697492848",
    "Account": "[account_id]",
    "Arn": "arn:aws:sts::[account_id]:assumed-role/ServiceRoleForGitLabPipeline/botocore-session-1697492848"
}

---

For the image build and push to ECR, I'm really not sure what I'm missing. Here is the job definition:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.12.1-debug
    entrypoint: [""]
  variables:
    AWS_EC2_METADATA_DISABLED: "true"
    AWS_SDK_LOAD_CONFIG: "true"
  before_script:
    - mkdir -p /kaniko/.docker ~/.aws
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
    - cat $AWS_CONFIG > ~/.aws/config
    - cat $AWS_CREDENTIALS > ~/.aws/credentials
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --target "lambda-ctx"
      --destination "${ECR_REPO_URI}:${CI_COMMIT_SHORT_SHA}"
      --skip-unused-stages=true
      --snapshot-mode=redo
  after_script:
    - cat ~/.ecr/log/ecr-login.log

The line that throws the error in the job:

error pushing image: failed to push to destination [account_id].dkr.ecr.us-east-1.amazonaws.com/lambda-wbxf:5a7d58ff: HEAD https://[account_id].dkr.ecr.us-east-1.amazonaws.com/v2/lambda-wbxf/manifests/5a7d58ff: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

And the output of the log file at ~/.ecr/log/ecr-login.log

time="2023-10-16T22:08:32Z" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-16T22:08:32Z" level=debug msg="Retrieving credentials" region=us-east-1 registry=[account_id] serverURL=[account_id].dkr.ecr.us-east-1.amazonaws.com service=ecr
time="2023-10-16T22:08:32Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=[account_id]
time="2023-10-16T22:08:32Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-16T22:08:32Z" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-16T22:08:32Z" level=debug msg="Retrieving credentials" region=us-east-1 registry=[account_id] serverURL=[account_id].dkr.ecr.us-east-1.amazonaws.com service=ecr
time="2023-10-16T22:08:32Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=[account_id]
time="2023-10-16T22:08:32Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"

---

It feels like the credential helper is not able to locate the credentials that are on the filesystem.

Any help on getting this squared away would be extremely helpful. Thanks!

[supergibbs]

Do you have Instance Metadata V2 (imdsv2) set to required? https://github.com/awslabs/amazon-ecr-credential-helper/issues/289