Persistent states across restarts

[frdmn]

Hi 👋

Great project!

It would be great if the last state of a probe/replica is stored somewhere persistent to make sure it doesn't trigger an obsolete notification (which was already sent when the service actually went down) again upon next start.

[valeriansaliou]

Hi there!

Thanks for the suggestion, that's indeed something I've thought about. In my use case, services report quickly to Vigil, so full recovery from a Vigil failure would only take 30 seconds maximum in this case, which is acceptable.

How would you implement a persistence layer? Would that be a serialized text file w/ all node states getting dumped every few minutes from memory, or would that be a SQLite DB (overkill imho).

I accept PRs on this.

[frdmn]

Hi @valeriansaliou!

good question - IMHO a file based storage with the serialized information would be sufficient enough.

Another quick and dirty workaround would be to "pause" the broadcast of notification after start-up for a certain time to allow nodes/replicas to enter the dead state again.

Would love to contribute but unfortunately I have not much experience with Rust (not "proper" programming in general) so I won't be much of a help here...

[andrikoz]

+1 for this