Ability to filter notifications

Background

Currently, to the best of my knowledge, nomad-toast sends notifications whenever the ModifyIndex of deployments or allocations changes. Even for a moderately busy nomad cluster this results in quite a large amount of notifications.

It would be very useful if it were possible for nomad-toast to filter and only notify on certain kinds of changes. This would allow nomad-toast to be used as a simple alerting tool (by only watching for failed allocations for example).

Suggestion

As far as I can tell there's no way in the nomad API to constrain the blocking query to only return certain kinds of allocations / deployments / etc.

I therefore think that nomad-toast would have to do the filtering itself. From a quick code read it seems like we could do this either during the runAllocationWachter type step or during the notifier step (as part of formatAllocationMessage).

As a start, maybe we can filter allocations against ones that have TaskStates which are Failed cf. API ref For deployments, maybe ones that have UnhealthyAllocs > 0?

If that makes sense, I'm happy to try my hand at this and send a PR

@ExalDraen I really like the suggestion here, I think it would be best to add this as a CLI arg that user can pass when running the allocation runner which potentially takes a list of allocation states to notify about. If you have the time to take a stab at this please feel free to, it would be appreciated. If you have any questions also, please reach out to me.

I'll have a think about the deployments, I am using/working on this currently so would like to just see how my current work plays before making a choice on those.

@jrasell Ok, I'll give it a shot. Some thoughts on how I plan to approach this below.

As far as I can tell from the Go API docs an allocation's overall status is represented by the ClientStatus field, which can take the values defined in the Constants section.

My plan is to add a filter against this as part of runAllocationWatcher and that check against the configured states.

A CLI parameter sounds a-ok for this, I'll just need to find a robust way of mapping the parameter to the internal representation, ideally with validation.

@jrasell I've created a PR with my first attempt, but I'd like some feedback. Could you please take a look?

@jrasell, could I ping you for an update on this? I just need a bit of feedback and then I think this can be merged