Enable to skip pruning by specified interval for every pruning class.

[keisuke-umezawa]

Motivation

We want to specify interval to skip pruning, so that we can fasten iterations. In most cases, we do not need to run pruning after each iteration, but currently we can not avoid it.

Description

Enable to skip pruning by specified interval for the certain PruningCallback classes.

• PruningCallback
  • [x] LightGBMPruningCallback
  • [ ] XGBoostPruningCallback

[VladSkripniuk]

TensorFlowPruningHook skips pruning based on run_every_steps argument. ChainerPruningExtension skips pruning based on pruner_trigger argument. I think we need a discussion to decide whether we need to reformat these two classes to add interval argument.

[toshihikoyanase]

I think we have an alternative approach to skip pruning. How about adding a method like BasePruner.is_target_step(step: int) -> bool so that the callbacks can query the timing of pruning to the pruners?

This is because the pruners have logic to skip pruning. If we add the interval argument to the callbacks, users may need to take care of the relationship between the logic and the interval argument.

For example, MedianPruner has the n_interval_steps argument to specify the interval in number of steps between the pruning checks. And SuccessiveHalvingPruner determines the pruning steps based on the arguments.

It may be confusing if users use both the interval argument and SuccessiveHalvingPruner because the pruning interval of SuccessiveHalvingPruner increases an exponential manner. So, IMO, the BasePruner.is_target_step method will improve usability.

I'm not fully sure if it covers actual use cases. So, if you have any comments and questions, please let me know.

c.f., https://github.com/optuna/optuna/pull/1161#issuecomment-625328649

[keisuke-umezawa]

This is because the pruners have logic to skip pruning. If we add the interval argument to the callbacks, users may need to take care of the relationship between the logic and the interval argument.

That is true. The interval should be managed by pruning class.

I summarized that what will use interval information.

1. pruning class. e.g. optuna.pruning.MedianPruner.
2. callback function. e.g. optuna.integration.KerasPruningCallback.
3. 3rd party function that control interval. e.g. chainer.training.triggers.IntervalTrigger.

My suggestions are as follows:

1. As @toshihikoyanase said, the interval information should be managed by pruners.
2. callback function will refer the interval from pruners.
3. We should align the interval both in 3rd party function and our callback.

[keisuke-umezawa]

@VladSkripniuk cc: @toshihikoyanase @hvy

Sorry for being late for updates.

I discussed it with the other developers, and we concluded it as follows:

1. The intervals for reporting and pruning should be configured separately.
2. Pruners should have the information of pruning interval, whereas callbacks should have the information of reporting interval.

Currently, I am implementing the above feature for lightgbm callback in #2490 .

[github-actions[bot]]

This issue has not seen any recent activity.

[github-actions[bot]]

This issue was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.

[github-actions[bot]]

This issue has not seen any recent activity.

[hvy]

https://github.com/optuna/optuna/pull/2490 has now been merged. The only integration callback that probably should adopt the same change, after which we can close this issue is XGBoostPruningCallback due to the usual large number of iterations.

[keisuke-umezawa]

@hvy Thank you for the comment. I updated the top description.

[g-votte]

Since this issue has been open for a long time, let me close once. Feel free to reopen upon necessity.