Failure handling

[tonmoy-saikia]

How does BOHB handle failures currently? AFAIK, they are just considered as bad configurations. It would be nice if specific exceptions (Eg: IO or network connectivity errors ) can be handled with a min number of retries.

[sfalkner]

Hey Tonmoy, sorry for the delay. I am not sure what kinds of exceptions you want to handle. You can always put a try-except statement into your worker's compute method and repeat the computation yourself. Catching network connectivity errors and retrying is a bit more tricky. Do you see those a lot, because I have not experiences those ever, but AFAIK, you run really long jobs, right?

[guysmoilov]

@sfalkner I would like to use BOHB for long running training jobs as well, on preemptible instances. I can't imagine that all workers will survive the whole optimization process. It would definitely be important to retry failed configurations at least a couple of times, and the high-budget jobs are especially likely to fail at some point due to longer running times. Come to think of it, the way to handle this best would probably be to have the Worker save periodic checkpoints to cloud storage, and be able to re-create the Worker if it fails and resume from that checkpoint, until it completes its allotted budget successfully. Of course, this means that TCP socket connectivity is NOT the way to find out that a Worker is permanently dead - you have to handle the situation where the same job is now resumed by a new Worker instance. So instead of sockets, it should be persistent event queues or DB entries or something. In effect, I guess this means getting rid of the NameServer as it is now and replacing it with a persistent cloud data structure to represent current state, which can be written and read by Workers and the Master.