litestream.io icon indicating copy to clipboard operation
litestream.io copied to clipboard

Published 20 hours ago • verified publisher icon benbjohnson

docs: s3 replication tweaks

Open mikeblum opened this issue 2 years ago • 3 comments

  • enable s3 bucket versioning
  • enable s3 bucket-level encryption with aws/s3 default keys

Hi Litestream!

Was setting up Litestream to replicate to my AWS account and figured I'd update the docs to include some S3 best practices. Let me know what you think and any notes.

Cheers, Mike

mikeblum avatar Jan 21 '23 22:01 mikeblum

@mikeblum I haven't used the bucket versioning or encryption before. Is there a trade-off to enabling either of them? Does the versioning make it so that you continue to pay for deleted files? Also, is it more complex to use the encryption?

benbjohnson avatar Jan 28 '23 18:01 benbjohnson

good questions @benbjohnson !

When bucket versioning is enabled you pay for new versions but deleting prior versions or the object completely will reduce your bill:

https://aws.amazon.com/s3/faqs/

I opted to use the out-of-the-box encryption key managed by AWS aws/s3. This way users don't need to manage their own KMS key (known as a CMK which costs $1/month/region otherwise but the default aws/s3 key is included). The only caveat here is you want to enable Bucket Key (which is the default) which makes calls to KMS much less frequently and therefor you save $$$:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-key.html

Given that many devs use a single bucket for their application (db, assets, etc) I figured it would be good to have their bucket configured against accidental deletion or exposing their data publicly.

I figured Litestream would make the necessary DELETE calls as most of the supported storage backends don't support versioning natively anyways.

mikeblum avatar Jan 28 '23 20:01 mikeblum

Also, is it more complex to use the encryption?

As an app developer the encryption is seamless / not noticeable. As an operator provisioning the bucket and IAM policy there is a canned policy for customer-managed keys but that won't be necessary when using default bucket encryption via aws/s3 managed key.

TIL AWS is doing this by default for new buckets as of January 5, 2023? https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html

mikeblum avatar Jan 29 '23 12:01 mikeblum