Cover local file backups to S3

[jlevy]

Cover some common approaches to backup filesystems to S3. See also #49.

Would be good to mention backup options/tools.

• https://github.com/zbackup/zbackup (deduplicating backups, inspired by rsync, in C++, analyzed here)
• https://github.com/restic/restic (deduplicating backups, in Go)
• https://github.com/borgbackup/borg (deduplicating backups, in Python, a fork of Attic with more active development)
• https://github.com/ncw/rclone (data sync to cloud)
• https://github.com/camlistore/camlistore

[VincentMarmiesse]

Hello,

Any news on this point?

[mclang]

Bacula is also tried and true option for doing backups. And maybe, even though this is about backing up to S3, Tarsnap should be mentioned as alternative, in case somebody wants to make backups outside AWS?

[mclang]

I was cleaning up my watch/starred/subscription list and found this old thread.

I'm not sure if this is needed anymore, but as an example, i'm currently backing up files into S3 using AWS-CLI. The bucket I use are setup to use encryption as follows:

1. Create S3 Bucket with following policy:

{
  "Version": "2012-10-17",
  "Id": "PutObjPolicy",
  "Statement": [
    {
      "Sid": "DenyIncorrectEncryptionHeader",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::<BUCKET-NAME>/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    },
    {
      "Sid": "DenyUnEncryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::<BUCKET-NAME>/*",
      "Condition": {
        "Null": {
          "s3:x-amz-server-side-encryption": "true"
        }
      }
    }
  ]
}

2. Create Bucket User with following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET-NAME>"
            ]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": [
                "arn:aws:s3:::<BUCKET-NAME>/*"
            ]
        }
    ]
}

3. Make backup using AWS-CLI:

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
/bin/aws s3 sync --no-follow-symlinks --sse AES256 "s3://<BUCKET-NAME>"