Timeout for an observer?

[lotsoffood]

Is there a method for setting a timeout period for an observer? For example, say the observer is a SQL database, and that database becomes inaccessible for a few minutes. Right now, the experiment simply fails. Is there an argument somewhere to avoid such a situation? Particularly for multi-day experiments.

[JarnoRFB]

You could try to wrap your SQLObserver in a QueueObserver. This does not have a timeout at all, but will just try to send events to the database forever. However, it should work around cases in which your backend becomes unavailable and comes back up later on.

[lotsoffood]

Thanks for the reply @JarnoRFB - what would be the easiest way to do what you're suggesting?

[JarnoRFB]

@lotsoffood Sorry for the delay. I finally wrote some documentation about the QueueObserver https://sacred.readthedocs.io/en/stable/observers.html#queue-observer. Let me know if that helps or what kind of other documentation would be useful.

[Chen-Cai-OSU]

Hello,

Is it possible to provide a small example to showcase the use of limiting the running time of an experiment in a certain amount of time? I see sacred.utils.TimeoutInterrupt but how should I use it?

[JarnoRFB]

@Chen-Cai-OSU I believe TimeoutInterrupt has to be raise from your experiments main function. If you have iteration is in your main function, you could check every once in a while if the experiments already took to long. If you do not have this possibility, maybe because you run a big chunk of computation or each iteration step is very long, then you could put that chunk in a separate thread and await the future of that thread as long as your desired timeout.

Something like this could be a starting point

from concurrent.futures import ThreadPoolExecutor, TimeoutError
from time import sleep
from sacred.utils import TimeoutInterrupt

def task():
    sleep(5)
    return "Task done"


with ThreadPoolExecutor(2) as excecutor:
    future = excecutor.submit(task)

    try:
        print(future.result(timeout=0.1))
    except TimeoutError:
        raise TimeoutInterrupt()

Where task represents the big computation you do in your experiment.

[Chen-Cai-OSU]

Thank you very much! I got the idea now.