[Bug]: aws_ecr_lifecycle_policy fails when using the aws:ResourceTag/${TagKey} IAM condition

[NFarrington]

Terraform Core Version

1.3.5

AWS Provider Version

4.40.0

Affected Resource(s)

• aws_ecr_repository
• aws_ecr_lifecycle_policy

Expected Behavior

When the AWS IAM principal used to apply the Terraform run is configured with the following IAM policy, terraform apply should successfully create a new aws_ecr_repository and aws_ecr_lifecycle_policy at the same time without generating an error.

Policy (click to expand)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:GetLifecyclePolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:ListTagsForResource",
                "ecr:DescribeImageScanFindings"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/Owner": "Example"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:DeleteRepository",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage",
                "ecr:PutLifecyclePolicy",
                "ecr:SetRepositoryPolicy",
                "ecr:GetRepositoryPolicy",
                "ecr:DeleteLifecyclePolicy"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/Owner": "Example"
                }
            }
        }

Actual Behavior

The aws_ecr_repository resource is successfully created, but the aws_ecr_lifecycle_policy fails to apply due to a lack of permissions.

A subsequent plan and apply results in the resource successfully being applied.

Relevant Error/Panic Output Snippet

Error: AccessDeniedException: User: arn:aws:iam::************:user/Terraform-example-docker-service is not authorized to perform: ecr:PutLifecyclePolicy on resource: arn:aws:ecr:us-east-2:************:repository/example because no permissions boundary allows the ecr:PutLifecyclePolicy action status code: 400, request id: 69324c33-f90e-4da0-aa4d-674f26a4dcb0

with aws_ecr_lifecycle_policy.main
on main.tf line 11, in resource "aws_ecr_lifecycle_policy" "main":
resource "aws_ecr_lifecycle_policy" "main" {

Terraform Configuration Files

Terraform Code (click to expand)

provider "aws" {
  region = "us-east-2"

  default_tags {
    tags = {
      Owner = "Example",
    }
  }
}

resource "aws_ecr_repository" "main" {
  name = "example"

  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_ecr_lifecycle_policy" "main" {
  repository = aws_ecr_repository.main.name

  policy = jsonencode({
    "rules" : [
      {
        "rulePriority" : 10,
        "description" : "Expire untagged images older than 7 days",
        "selection" : {
          "tagStatus" : "untagged",
          "countType" : "sinceImagePushed",
          "countUnit" : "days",
          "countNumber" : 7
        },
        "action" : {
          "type" : "expire"
        }
      },
    ]
  })
}

Steps to Reproduce

1. Create a user with the IAM policy documented in the Expected Behavior section above attached to it. Additionally, attach the same policy as the user's permissions boundary.
2. Create an access key for the user and configure the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
3. Plan and apply the provided Terraform code from the previous section.
4. Receive the AccessDeniedException documented in the Relevant Error/Panic Output Snippet section above.
5. Plan and apply Terraform again, resulting in success.

Debug Output

No response

Panic Output

No response

Important Factoids

This is a very similar issue to the one described in https://github.com/hashicorp/terraform-provider-aws/issues/27028 (which I have also encountered), however, I've raised a separate issue as there are a few significant differences, e.g. this issue uses two separate resources, with only the lifecycle policy failing. The referenced issue occurs solely within a single resource.

If I add a sleep between the creation of the ECR repository and the lifecycle policy, the issue still occurs, which seems to suggest this isn't an eventual consistency issue, which was the cause suggested for https://github.com/hashicorp/terraform-provider-aws/issues/27028. In the example provided below, waiting 3 minutes between the creation of the ECR repository and the creation of the lifecycle policy still results in the same error. It seems you have to replan Terraform after the repository has been created in order for it to be created successfully.

Terraform Code With Sleep (click to expand)

resource "aws_ecr_repository" "main" {
  name = "example"

  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "time_sleep" "wait_for_ecr_repository" {
  depends_on = [aws_ecr_repository.main]

  create_duration = "180s"
}

resource "aws_ecr_lifecycle_policy" "main" {
  depends_on = [time_sleep.wait_for_ecr_repository]

  repository = aws_ecr_repository.main.name

  policy = jsonencode({
    "rules" : [
      {
        "rulePriority" : 10,
        "description" : "Expire untagged images older than 7 days",
        "selection" : {
          "tagStatus" : "untagged",
          "countType" : "sinceImagePushed",
          "countUnit" : "days",
          "countNumber" : 7
        },
        "action" : {
          "type" : "expire"
        }
      },
    ]
  })
}

This issue does not occur if you use ecr:ResourceTag/${TagKey} instead of aws:ResourceTag/${TagKey}. However, as far as I know aws:ResourceTag/${TagKey} is supposed to be a completely viable option here and there's no documented difference between the condition keys, so there's no documented reason for them to behave differently.

Since there's no equivalent condition for CloudWatch (i.e. there's no logs:ResourceTag/${TagKey} or cloudwatch:ResourceTag/${TagKey} condition), it's not possible to determine whether this is caused by the same issue as https://github.com/hashicorp/terraform-provider-aws/issues/27028.

References

Actions, resources, and condition keys for Amazon Elastic Container Registry - Service Authorization Reference

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[danielkza]

I'm observing similar behaviour using boto3 and no Terraform whatsoever, so it might be some AWS issue.