alertmanager icon indicating copy to clipboard operation
alertmanager copied to clipboard

mute_time_intervals should be inherited to sub routes

Open LeoQuote opened this issue 3 years ago • 1 comments

What did you do? I set up a route to mute all warning alert outside work days, and several sub routes to route these alerts to different receivers.

What did you expect to see? I was expecting this alert could be silenced in all sub routes

What did you see instead? Under which circumstances? the alert was sent regardless of the mute intervel setting Environment

  • System information:

    insert output of uname -srm here

  • Alertmanager version:

    insert output of alertmanager --version here (repeat for each alertmanager version in your cluster, if relevant to the issue) v0.23.0

  • Prometheus version:

    insert output of prometheus --version here (repeat for each prometheus version in your cluster, if relevant to the issue)

  • Alertmanager configuration file:

insert configuration here
  • Prometheus configuration file:
insert configuration here (if relevant to the issue)
  • Logs:
insert Prometheus and Alertmanager logs relevant to the issue here

I'm aware the document said:

# Times when the route should be muted. These must match the name of a
# mute time interval defined in the mute_time_intervals section. 
# Additionally, the root node cannot have any mute times.
# When a route is muted it will not send any notifications, but
# otherwise acts normally (including ending the route-matching process
# if the `continue` option is not set.)
mute_time_intervals:
  [ - [<string>](https://www.prometheus.io/docs/alerting/latest/configuration/#string) ...]

but I think these rules should be inherited, this can make the config shorter and easier to read, otherwise I have to set mute time for every single route.

LeoQuote avatar Feb 14 '22 05:02 LeoQuote

I also found the current behavior to be inconvenient. And the documentation says that children inherit optional parameters from the parents in the routing tree, so this behavior is also not obvious.

mpol avatar Jul 11 '22 13:07 mpol

I haven't checked this, but it sounds like a bug to me - /cc @benridley @simonpasquier

juliusv avatar Aug 24 '23 14:08 juliusv

Agreed that it doesn't look very consistent and I see the point of having the same mute times being applied to all subroutes . One reason though to explain the current behavior: unlike group_interval et al., an empty mute_time_intervals is valid so if time intervals were inherited by child routes, it would be hard to distinguish between "apply from the parent" and "don't apply from the parent".

simonpasquier avatar Aug 30 '23 09:08 simonpasquier

Thanks @juliusv - I've opened #3538 to fix this. It is indeed a bug, I forgot to check for empty config on child routes before overwriting parent config, so it overwrote the parent config every time.

benridley avatar Sep 29 '23 00:09 benridley