mimir
mimir copied to clipboard
Alertmanager: Support running without PVC
With some small tweaks, we can run Alertmanager without PVCs.
The only state on the disk is:
- Template files
- These are a) tiny and b) can be trivially recreated from the tenant configuration
- I think the only reason they are on disk at all is to pass them to the upstream Prometheus code, so perhaps we can some up with an alternative.
- Silences/Notification state snapshots
- Since we introduced sharding, this state is actually borderline useless, as the silence/notification state is fetched from replicas or object storage at startup.
Describe the solution you'd like
- Configuration option to disable of writing and reading the state snapshots from local disk
- This configuration option does not need to be long lived, and could be removed in a version or two time, and be the default.
- Tweak jsonnet/helm to allow configuring the alertmanager data volume as ephemeral instead of persistent, and reduce the default size to something much smaller (100Gi will be overkill for template files).