Two-way replication configuration does not complete

[sbrinkerhoff]

Description

I am attempting to configure two-way S3 bucket replication, and receive an error that InvalidRequest: Destination bucket must have versioning enabled. when doing so. Immediately issuing a subsequent terraform apply completes with no errors and a successful configuration.

Alternatively creating the buckets and then adding the replication configuration in afterwards works fine with no intermediate error.

This is similar to #42, except the issue references in #42 was resolved in upstream.

• [X] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

v3.3.0

• Terraform version: Terraform 1.1.9

• Provider version(s): Terraform v1.1.9 on darwin_amd64

• provider registry.terraform.io/gavinbunney/kubectl v1.14.0
• provider registry.terraform.io/hashicorp/aws v4.24.0
• provider registry.terraform.io/hashicorp/cloudinit v2.2.0
• provider registry.terraform.io/hashicorp/helm v2.6.0
• provider registry.terraform.io/hashicorp/kubernetes v2.10.0
• provider registry.terraform.io/hashicorp/random v3.3.2
• provider registry.terraform.io/hashicorp/time v0.7.2
• provider registry.terraform.io/hashicorp/tls v3.4.0

Reproduction Code [Required]

locals {
  asset_primary_name   = "assets-${local.environment}-${local.regions.primary}-${random_pet.asset_random.id}"
  asset_secondary_name = "assets-${local.environment}-${local.regions.secondary}-${random_pet.asset_random.id}"
}

resource "random_pet" "asset_random" {
}

resource "aws_iam_role_policy" "s3_policy" {
  name = "asset-${local.environment}-policy"
  role = var.arn
  policy = data.aws_iam_policy_document.document.json
}

data "aws_iam_policy_document" "document" {
  statement {
    sid    = "1"
    effect = "Allow"
    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging",
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
      "s3:GetReplicationConfiguration",
      "s3:ListBucket"
    ]

    resources = [
      "${module.assets_primary.s3_bucket_arn}/*",
      "${module.assets_secondary.s3_bucket_arn}/*"
    ]
  }
}

module "assets_primary" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.3.0"

  bucket = local.asset_primary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = data.aws_kms_key.key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = var.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_secondary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = data.aws_kms_key.key-us-west-2.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

module "assets_secondary" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.3.0"
  providers = {
    aws = aws.us-west-2
  }

  bucket = local.asset_secondary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = data.key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = var.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_primary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = data.aws_kms_key.key.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

Steps to reproduce the behavior:

Are you using workspaces?

Yes.

Have you cleared the local cache (see Notice section above)?

Yes.

List steps in order that led up to the issue you encountered

Terraform apply.

Expected behavior

Two buckets with replication would be created with a single terraform apply without error.

Actual behavior

│ Error: error creating S3 replication configuration for bucket (assets-dev-us-east-1-enjoyed-koala): InvalidRequest: Destination bucket must have versioning enabled.
│ 	status code: 400, request id: 3SWPGQ12GXV1TS6S, host id: 0L6HAwm2PqjnL6jvM7dTX42cJvzfcUePCRWoDn9jw1RAqSyeUTbMuMvOAFPS18gL5Jr7MoS9qak=
│
│   with module.assets_primary.aws_s3_bucket_replication_configuration.this[0],
│   on .terraform/modules/assets_primary/main.tf line 354, in resource "aws_s3_bucket_replication_configuration" "this":
│  354: resource "aws_s3_bucket_replication_configuration" "this" {
│

Terminal Output Screenshot(s)

See above.

Additional Detail

This appears to be a race condition of sorts --

• If I create the two buckets, and then add the replication configuration in, it adds just fine.
• If I run terraform apply, let it fail, and then terraform apply again it works just fine.
• Checking the bucket immediately after the first fail, the secondary bucket does have versioning enabled.