Delete EBS snapshots tagged as Prometheus and created by velero

[vijay-veeranki]

Background

We received an alert related to EBS Active snapshots, where the limit is 100000 and the current usage is 80205.

Looking further noticed that we have snapshots from “2019/05/28”

https://mojdt.slack.com/archives/C8QR5FQRX/p1658228620025229

Proposed user journey

Agree with the team and delete prometheus snapshots older than 1 month

Approach

Create a new lifecycle EBS snapshot policy to delete snapshots tagged as Prometheus and created by velero older than 1 month

Also, create a script running in a pipeline to delete test cluster snapshots.

snapshots_to_delete=$(aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[?StartTime<=2019-06-15`].SnapshotId' --output text) echo "List of snapshots to delete: $snapshots_to_delete"

actual deletion

for snap in $snapshots_to_delete; do aws ec2 delete-snapshot --snapshot-id $snap done get prometheus tagged snapshots: aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[?!not_null(Tags[?Key ==prometheus`].Value)]'

Which part of the user docs does this impact

Questions / Assumptions

Definition of done

• [ ] EBS snapshots tagged as Prometheus and created by velero deleted older than 1 month
• [ ] Test cluster snapshots deleted older than 1 month
• [ ] New lifecycle EBS snapshot policy to delete snapshots tagged as Prometheus
• [ ] Pipeline to delete test snapshorts

Reference

How to write good user stories