Need a way to clear old states

[phobologic]

trafficstars

Currently once something is recorded in the state database it's there forever. An example of when this is bad:

• You launch an instance and configure it to be monitored in NYMMS
• Eventually you replace that instance with a new instance and update your NYMMS config. Now you're monitoring the new instance, but not the old.

In that case the old state will remain in the state backend, even though it won't be used for anything. It might be worth having the scheduler or something similar clean up old states - the only issue with that is that we'd have to find some way to deal with passive monitors since the scheduler has no concept of passive monitors (the passive monitors just send in results themselves). That may be a bug in and of itself.

[sethlindberg]

I don't know much about how nymms handles states, but is there some thought put into how to monitor instances that may be autoscaled (like an application that doubles its app server load during periods of heavy use and then removes those instances)

this is an issue at least with old versions of Sensu: removing a host or even a check can be a laborious process that way.

[phobologic]

Yes - though I've left it sort of loose so that people can do it anyway that they want. In the end it comes down to this:

In your nodes.yaml you can include files, including a directory of files that define new nodes. For example:

!include /etc/nymms/nodes/*.yaml

That would include every *.yaml file in that directory into your nodes definition. So all you would have to do is populate that directory (with many files, or one big one - I currently use one big one) and then restart the scheduler.

[phobologic]

BTW, there may be room for a more generic way of doing this and I might write a daemon to do so at some point - but there are many ways that people may want to control adding and removing of nodes, so I've just given people the tools in the config for now.