operations icon indicating copy to clipboard operation
operations copied to clipboard

Cross Account S3 Replication for openstreetmap-gps-traces

Open Firefishy opened this issue 3 years ago • 2 comments

Setup cross account S3 replication for openstreetmap-gps-traces to an S3 bucket in an AWS backup account.

Firefishy avatar Sep 07 '22 20:09 Firefishy

Linked issue: https://github.com/openstreetmap/operations/issues/744

Firefishy avatar Sep 07 '22 20:09 Firefishy

Blocked by: https://github.com/openstreetmap/operations/issues/746

Firefishy avatar Sep 07 '22 20:09 Firefishy

Unblocked now that #746 is closed

grischard avatar Dec 15 '22 19:12 grischard

S3 buckets:

  • openstreetmap-gps-images (source) → openstreetmap-gps-images-replicate (destination)
  • openstreetmap-gps-traces (source) → openstreetmap-gps-traces-replicate (destination)

Additional requirements:

  • Replication configuration with rule "blank" filter prefix and destination storage class
  • Source role which allows S3 to assume.
  • IAM policy attached to above role with permissions which allows replication.
  • Destination S3 bucket versioning enable
  • Destination S3 bucket private ACL
  • Destination S3 bucket access policy (deny)
  • Destination S3 bucket lifecycle rules (deleted objects?)
  • Destination S3 bucket intelligent tiering and with configuration allowing archiving?
  • S3 Batch Replication configuration (existing objects or failed)

Firefishy avatar Jul 06 '23 14:07 Firefishy

Terraform example: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_replication_configuration#example-usage

Firefishy avatar Jul 06 '23 14:07 Firefishy

openstreetmap-gps-images is now replication across account. More to follow.

Firefishy avatar Aug 11 '23 02:08 Firefishy

The bucket is now fully in-sync to a backup bucket in EU-North-1. The bucket lives in another dedicated backups account.

Firefishy avatar Aug 17 '23 15:08 Firefishy