mimir icon indicating copy to clipboard operation
mimir copied to clipboard

Alertmanager: Support running without PVC

Open stevesg opened this issue 2 years ago • 0 comments

With some small tweaks, we can run Alertmanager without PVCs.

The only state on the disk is:

  • Template files
    • These are a) tiny and b) can be trivially recreated from the tenant configuration
    • I think the only reason they are on disk at all is to pass them to the upstream Prometheus code, so perhaps we can some up with an alternative.
  • Silences/Notification state snapshots
    • Since we introduced sharding, this state is actually borderline useless, as the silence/notification state is fetched from replicas or object storage at startup.

Describe the solution you'd like

  • Configuration option to disable of writing and reading the state snapshots from local disk
    • This configuration option does not need to be long lived, and could be removed in a version or two time, and be the default.
  • Tweak jsonnet/helm to allow configuring the alertmanager data volume as ephemeral instead of persistent, and reduce the default size to something much smaller (100Gi will be overkill for template files).

stevesg avatar Feb 08 '23 12:02 stevesg