Rüdiger Busche

You could try to wrap your SQLObserver in a [QueueObserver](https://github.com/IDSIA/sacred/blob/master/sacred/observers/queue.py). This does not have a timeout at all, but will just try to send events to the database forever. However,...

@lotsoffood Sorry for the delay. I finally wrote some documentation about the QueueObserver https://sacred.readthedocs.io/en/stable/observers.html#queue-observer. Let me know if that helps or what kind of other documentation would be useful.

@Chen-Cai-OSU I believe `TimeoutInterrupt` has to be raise from your experiments `main` function. If you have iteration is in your main function, you could check every once in a while...

Hi, I absolute understand the frustration with an observer crashing in a long running experiment and we definitely need to endeavor to make it a lot more fault tolerant! However,...

Wow I did not expect an experiment to have so many epochs. In this case we need to consider saving metrics in a different form as you suggested. Many distributing...

@SumNeuron Yeah that was basically the idea. I talked to some people that said that they use the FileStorageObserver, because it is easier to collaborate on... However, it is easier...

@SumNeuron yes if you are working on a project alone that is definitely easier and cheaper. But for larger projects with many people working on the same thing you want...

Thanks for the ideas!

Thanks for reporting! I would at least think that you should get around this by explicitly injecting individually seeded random number generators into your captured/main functions as described in https://sacred.readthedocs.io/en/latest/randomness.html#special-arguments....

@decodyng I would be definitely delighted by an S3 observer. I believe it is something that many people could make good use of!