CodeBuild push to ECR doesn't work

[tim-finnigan]

Original issue: https://github.com/aws/aws-cli/issues/7231 by @AMB-Informacio

Describe the bug

It's the same behaviour as the described in:

https://github.com/aws/aws-cli/issues/7102

A buildspec.yml file describes to build a docker image and to push it to ECR but the Docker push don't work, retrying to do that and finally fails.

There was answered as a problem with the artifacts and the image-definitions.json file. But that's happening because the previous step "docker push" fails unexpectedly, and that is the real problem.

Expected Behavior

Docker image pushed to ECR

Current Behavior

For any reason docker push is retrying and then an EOF appears.

[Container] 2022/08/30 06:32:01 Running command docker push $REPOSITORY_URI:latest
--
723 | The push refers to repository [XXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/gitoccupacy-backend-service]
724 | 6a37c076ad0a: Preparing
725 | 838351f683ff: Preparing
726 | f00576e66739: Preparing
727 | 845237c43dbf: Preparing
[...]
1069 | 845237c43dbf: Retrying in 1 second
1070 | 94a688c21a1d: Retrying in 1 second
1071 | EOF
1072 |  
1073 | [Container] 2022/08/30 06:32:51 Command did not exit successfully docker push $REPOSITORY_URI:latest exit status 1
1074 | [Container] 2022/08/30 06:32:51 Phase complete: POST_BUILD State: FAILED

Reproduction Steps

Commit or retry to build project with this buildspec.yml file:

version: 0.2

env:
  secrets-manager:
    DOCKERHUB_LOGIN: "dockerhub:login"
    DOCKERHUB_PASSWORD: "dockerhub:password"

batch:
  fast-fail: false

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws --version
      - REPOSITORY_DOMAIN_NAME=$(echo $AWS_ACCOUNT_ID'.dkr.ecr.'$AWS_DEFAULT_REGION'.amazonaws.com')
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $REPOSITORY_DOMAIN_NAME
      - REPOSITORY_URI=$(echo $REPOSITORY_DOMAIN_NAME'/'$IMAGE_REPO_NAME)
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
      - echo Logging in to DockerHub
      - docker login --username $DOCKERHUB_LOGIN --password $DOCKERHUB_PASSWORD
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build --target release --file './backend-service/Dockerfile' -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker images...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Writing image definitions file...
      - printf '[{"name":"backend","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
artifacts:
    files: imagedefinitions.json

Possible Solution

No response

Additional Information/Context

No response

CLI version used

code build version: 0.2

Environment details (OS name and version, etc.)

code build version: 0.2

[AMB-Informacio]

Thanks @tim-finnigan !

[suratpyari]

did anyone find the fix?

[yingchangwu]

This is a issue related to permission on your IAM role for your build server / pipeline, the role associated with your pipeline / build server doesn't have permission to push to ecr.

[nerub1an]

So the solution is to add permissions from the ECR side, not IAM of CodeBuild role

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowProdEks",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::***:root"
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ]
    }
  ]
}

[leoherran-aws]

The answer above is correct, the necessary push permissions are required in both the CodeBuild service IAM role and the ECR repository policy. Docker unfortunately doesn't leave helpful error messages when these permissions are insufficient. We are working on updating our documentation to better communicate this requirement.

[atazangene]

something that is interesting is that I can push to ECR with github action without ECR repo permission, but I can not do that with aws code build! how that is possible ?

[rajrules21]

I had the same issue and instead of giving permissions from ECR end, i wrote commands to create the ECR repository and then my code build was successful. The ECR repo needs to be already existing for the post build commands to work.