Emails should be per-cycle, not per-rule

[mperham]

When something breaks, it's often the case that several rules will break at the same time. Each cycle's email should summarize all the rules that are broken and not a single email per rule. This summary allows the reader to see everything broken at a glance and possibly understand the root cause.

[felixbuenemann]

That's a great idea!

[felixbuenemann]

I've had a case were redis didn't properly restart after a package upgrade and consequently a lot of other services failed and I was bombarded with thousands of emails by Inspeqtor, so some kind of throttling/digest would be a great idea, especially because process id changes always cause alerts and can very quickly add up. I could probably mitigate the spamming by setting respawn limits on the upstart services, but it'll still generate a burst of messages until those kick in.

Maybe if emails would be per cycle digests, there could be an extra global setting to only send mails every n cycles for a simple form of throttling.

[nick-desteffen]

This is a great idea. We use Inspeqtor for monitoring and restarting our Sidekiq processes. Whenever the memory threshold is reached we get 4 emails, ☹️ . During traffic spikes our inboxes are flooded.