Live information not updated if the observer failed mid-run

[vnmabus]

I have noticed that, if an observer failed mid-run (for example, if the Mongo database became temporarily unavailable) the heartbeats stop occurring. If the problem is solved when the experiment finishes, then the result is saved and the run is marked as completed BUT the live information that is updated as part of the heartbeat event does not receive a last update, so it is incomplete.

Ideally, I would expect that Sacred retries the heartbeat event after some time has passed, so if the problem was transient the live information continues updating. If that is not desirable for whatever reason, at least I would expect that Sacred updates this information as part of the final save procedure.

[Qwlouse]

Good point, thank you for bringing it up! We should definitely make sure the complete information is stored on final save. Resuming the heartbeat after some time also sounds like a good idea, though that will require a bit more thought. Especially because of error handling which should be improved anyways (see #314).

[Gonzalo933]

I'm also experiencing this issue, any progress on it?