Attach EBS volumes to etcd nodes for persistence

[adambom]

If you ever lose your etcd cluster for whatever reason, or if you should ever need to restart it, you should be able to recover your state. Mentioned in this issue: https://github.com/kz8s/tack/issues/75

I have a module that will automatically snapshot the drives. If that's at all of interest, happy to commit that as well.

[rimusz]

the snapshot module PR would be awesome too

[adambom]

@rimusz Ok I've updated this to take automatic snapshots of ALL ebs volumes associated with the cluster at a rate of once every two hours. This includes the etcd volumes as well as any persistent volumes created by Kubernetes.

Happy to split this into two PR's if you'd prefer to review it that way.

For some background, originally I tried doing the snapshotting using only cloudwatch, but it wouldn't work without some manual configuration, so I ended up following the lambda approach described here. This also gives us the added benefit of backing up the dynamically generated volumes as well since it's based on tags.

[wellsie]

ty @adambom !!

No need to split into two PRs - this one is fine.

If you wanted to add a few notes to the readme, that would be great - np if not, I will add.

[adambom]

@wellsie will do

[adambom]

Done