Ability to filter notifications

[ExalDraen]

Background

Currently, to the best of my knowledge, nomad-toast sends notifications whenever the ModifyIndex of deployments or allocations changes. Even for a moderately busy nomad cluster this results in quite a large amount of notifications.

It would be very useful if it were possible for nomad-toast to filter and only notify on certain kinds of changes. This would allow nomad-toast to be used as a simple alerting tool (by only watching for failed allocations for example).

Suggestion

As far as I can tell there's no way in the nomad API to constrain the blocking query to only return certain kinds of allocations / deployments / etc.

I therefore think that nomad-toast would have to do the filtering itself. From a quick code read it seems like we could do this either during the runAllocationWachter type step or during the notifier step (as part of formatAllocationMessage).

As a start, maybe we can filter allocations against ones that have TaskStates which are Failed cf. API ref For deployments, maybe ones that have UnhealthyAllocs > 0?

If that makes sense, I'm happy to try my hand at this and send a PR