mlr3
mlr3 copied to clipboard
timeout for resample or benchmark
Besides a timeout for individual learners, the user may want to have a timeout for resample / benchmark. This could, in particular, be used inside tuners that want to limit the runtime of eval_batch calls. A way to implement this would be to change the learner$timeout
before each call. (change learner$timeout["train"]
to the minimum of its previous value and remaining resample / benchmark time before training, change learner$timeout["predict"]
to min of remaining time and its previous value after training and before predicting; this would mean that a learner that trains fast gets more time for prediction etc.)
additionally, maybe we want to give the user an option to set a total timeout for learners instead of setting train_timeout and predict_timeout. Internally prediction timeout should then be the total timeout minus the time taken for training.