feat(s3): replicating objects

[badmintoncryer]

Issue # (if applicable)

Closes #1680.

Reason for this change

AWS S3 supports configuring object replication , but the s3.Bucket construct does not support it.

Description of changes

Added replicationRules to BucketProps.

Replication configuration version

There are two versions of replication configuration. This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC).

To use V2 replication configuration, this PR explicitly specifies Filter.Prefix property.

        const prefix = rule.prefixFilter ?? '';
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

V2 replication configuration has some restriction:

• Must specify DeleteMarkerReplication

ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

• Must specify Priority

ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

These restriction is not documented but there are some posts about these points.

• https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets

To resolve these problems,I made the priority required and explicitly set the deleteMarkerReplication.

       const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

        return {
          id: rule.id,
          priority: rule.priority,
          status: 'Enabled',
          destination: {
            bucket: rule.destination.bucket.bucketArn,
            account: rule.destination.account,
            storageClass: rule.storageClass?.toString(),
            accessControlTranslation: rule.destination.accessControlTransition ? {
              owner: 'Destination',
            } : undefined,
            encryptionConfiguration: rule.kmsKey ? {
              replicaKmsKeyId: rule.kmsKey.keyArn,
            } : undefined,
            replicationTime: rule.replicationTimeControl !== undefined ? {
              status: rule.replicationTimeControl ? 'Enabled' : 'Disabled',
              time: {
                minutes: 15,
              },
            } : undefined,
            metrics: rule.replicationTimeControlMetrics !== undefined ? {
              status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled',
              eventThreshold: {
                minutes: 15,
              },
            } : undefined,
          },
          filter,
          // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication,
          // CDK explicitly set the deleteMarkerReplication if it is undefined.
          deleteMarkerReplication: {
            status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled',
          },
          sourceSelectionCriteria,
        };

IAM permission

There is a documentation to setup IAM permissions for service role.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource":"arn:aws:s3:::DST-BUCKET/*"
      }
   ]
}

However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation.

Generated Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SRC-BUCKET",
                "arn:aws:s3:::SRC-BUCKET/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms",
                        "aws:kms:dsse",
                        "AES256"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::DST-BUCKET/*"
            ]
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": "s3.ap-northeast-1.amazonaws.com",
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::SRC-BUCKET/*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga"
            ]
        },
        {
            "Action": [
                "kms:Encrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "s3.ap-northeast-1.amazonaws.com"
                    ],
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::DST-BUCKET*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga"
            ]
        }
    ]
}

I adopted the policy from the document. I look forward to hearing your thoughts on this matter.

Description of how you validated changes

Added both unit and integ tests.

Checklist

• [x] My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[shikha372]

Hi @badmintoncryer , Thank you for submitting this PR, as this feature is adding some additional pemissions for s3, we'll need to have an internal security review while we go through this PR. Will keep you posted !!

[shikha372]

Hi @badmintoncryer, to update had the first security review for this PR, currently concern is around populating correct permissions for opt-in regions, will let you know if any changes are required to address it.

[badmintoncryer]

@rix0rrr Thank you for your review! I've addressed all of your comments.

[shikha372]

Hello @badmintoncryer , Apologies to keep you waiting, have completed the feature review for this PR. Current ask is if we can add integ test for bidirectional replication and for cross-account replication(if possible) as well before final approval.

[badmintoncryer]

@shikha372 Thank you for your information.

add integ test for bidirectional replication

What is the meaning of bidirectional replication? Are you suggesting that we prepare Bucket A and Bucket B, and designate each other as their respective replication destinations?

　cross-account replication(if possible)

I don't have two AWS account...　Would it be better to create a new account and run the tests?

[shikha372]

What is the meaning of bidirectional replication? Are you suggesting that we prepare Bucket A and Bucket B, and designate each other as their respective replication destinations?

• Yes, since this can also be supported using new feature, we'll need a integ test for this case.

I don't have two AWS account...　Would it be better to create a new account and run the tests?

• Got it, in that case I will help adding integ test for this one

[badmintoncryer]

I considered bidirectional replication. First, I added an addReplicationRules() method to allow the destination bucket to be set lazily.

  public addReplicationRules(rules: ReplicationRule[]): void {
    this.replicationRules.push(...rules);
  }

However, it seems that the circular reference between the source and destination buckets cannot be resolved.

Do you have any suggestions on the implementation?

[shikha372]

I considered bidirectional replication. First, I added an addReplicationRules() method to allow the destination bucket to be set lazily.

  public addReplicationRules(rules: ReplicationRule[]): void {
    this.replicationRules.push(...rules);
  }

However, it seems that the circular reference between the source and destination buckets cannot be resolved.

Do you have any suggestions on the implementation?

Thanks @badmintoncryer , working on adding cross-account integ test will give it a try at my end for bi-directional.

[badmintoncryer]

Thank you very much @shikha372

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.48%. Comparing base (717d91d) to head (f543e0f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #30966   +/-   ##
=======================================
  Coverage   81.48%   81.48%           
=======================================
  Files         226      226           
  Lines       13768    13768           
  Branches     2416     2416           
=======================================
  Hits        11219    11219           
  Misses       2271     2271           
  Partials      278      278           

Flag	Coverage Δ
suite.unit	81.48% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	80.89% <ø> (ø)
packages/aws-cdk-lib/core	82.10% <ø> (ø)

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergifyio update

[mergify[bot]]

update

✅ Branch has been successfully updated

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify[bot]]

update

✅ Branch has been successfully updated

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[shikha372]

Hi @badmintoncryer , tried to merge it yesterday but I think there is unit test coverage missing for new addReplication method, I can get back to it next week

[badmintoncryer]

Thank you very much @shikha372 . Looking forward to the updates!

[badmintoncryer]

@mergify refresh

[mergify[bot]]

refresh

✅ Pull request refreshed

[badmintoncryer]

@mergify update

[mergify[bot]]

update

✅ Branch has been successfully updated

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify[bot]]

update

✅ Branch has been successfully updated

[mergify[bot]]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify[bot]]

update

✅ Branch has been successfully updated