mlr3 icon indicating copy to clipboard operation
mlr3 copied to clipboard

timeout for resample or benchmark

Open mb706 opened this issue 3 years ago • 1 comments

Besides a timeout for individual learners, the user may want to have a timeout for resample / benchmark. This could, in particular, be used inside tuners that want to limit the runtime of eval_batch calls. A way to implement this would be to change the learner$timeout before each call. (change learner$timeout["train"] to the minimum of its previous value and remaining resample / benchmark time before training, change learner$timeout["predict"] to min of remaining time and its previous value after training and before predicting; this would mean that a learner that trains fast gets more time for prediction etc.)

mb706 avatar Nov 04 '20 17:11 mb706

additionally, maybe we want to give the user an option to set a total timeout for learners instead of setting train_timeout and predict_timeout. Internally prediction timeout should then be the total timeout minus the time taken for training.

mb706 avatar Nov 04 '20 17:11 mb706