FRAMEWORK SETUP UPDATE: set up the framework flaml's special requirements for automlbenchmark

[qingyun-wu]

• FLAML reorganized its default requirements. Some of the dependencies are no longer installed by default but are still needed to run experiments in automlbenchmark. We moved those ones into [benchmark]

[PGijsbers]

Thanks for the update! I have some additional questions to figure out how to best integrate this change into the automl benchmark. I noticed this installs the following extra dependencies:

catboost>=0.26
psutil==5.8.0
xgboost==1.3.3

1. Are these then automatically used when installed?

2. Currently some other frameworks also have specific benchmark modes (e.g. autogluon), but these are set through hyperparameters of the AutoML framework which means you can run both benchmark and non-benchmark modes from within this benchmark tool. If the extra packages are automatically used, would there still be a way to turn that off through hyperparameters?

3. I noticed psutil in the benchmark requirements. Is the default installation incapable of following benchmark constraints now (e.g. duration or cores)?

[qingyun-wu]

Thanks for the update! I have some additional questions to figure out how to best integrate this change into the automl benchmark. I noticed this installs the following extra dependencies:

catboost>=0.26
psutil==5.8.0
xgboost==1.3.3

1. Are these then automatically used when installed?
2. Currently some other frameworks also have specific benchmark modes (e.g. autogluon), but these are set through hyperparameters of the AutoML framework which means you can run both benchmark and non-benchmark modes from within this benchmark tool. If the extra packages are automatically used, would there still be a way to turn that off through hyperparameters?
3. I noticed psutil in the benchmark requirements. Is the default installation incapable of following benchmark constraints now (e.g. duration or cores)?

Hi @PGijsbers, Thank you for the questions. Please find my answers below and let me know if you have further questions or suggestions.

1. The three packages will be automatically used when installed (with the [benchmark] option).
2. The use of catboost and xgboost can be turned off through our estimator_list parameter. The use of psutil once installed cannot be turned off through a hyperparameter for now. But we can add one if necessary.
3. The default installation is capable of the benchmark's constraints. 'psutil' is added for memory control.

(F.Y.I. @sonichi)

[PGijsbers]

In principle, we expect modes to be configurable for the end-user not just through what is installed and what isn't but also through one hyperparameter that expresses the user intent. For example, autogluon has "best_quality" and "fast_inference" modes, and mljar has "compete" and "interpretability" modes, to name a few. Inferring the mode through the installed dependencies means users (and AMLB) can not use the same installed environment without "expert knowledge" of exactly what the difference between the modes is (i.e., the user would now need to know catboost and xgboost are not part of the default behavior, and should be turned off through estimator_list). Would you consider a hyperparameter which configures flaml based on user intent (i.e., to distinguish at least "normal use" and benchmark use)?

[qingyun-wu]

In principle, we expect modes to be configurable for the end-user not just through what is installed and what isn't but also through one hyperparameter that expresses the user intent. For example, autogluon has "best_quality" and "fast_inference" modes, and mljar has "compete" and "interpretability" modes, to name a few. Inferring the mode through the installed dependencies means users (and AMLB) can not use the same installed environment without "expert knowledge" of exactly what the difference between the modes is (i.e., the user would now need to know catboost and xgboost are not part of the default behavior, and should be turned off through estimator_list). Would you consider a hyperparameter which configures flaml based on user intent (i.e., to distinguish at least "normal use" and benchmark use)?

Hi @PGijsbers, thank you for your suggestion. I understand that adding a mode to configure the expected behavior is a good practice and we will add the modes in that way when we do have different choices of modes in the future. But this this benchmark version is needed because of the following motivation: we want to minimize the installed dependencies by the default flaml (we received requests of reducing mandatory dependencies). In other words, the installation of dependencies is exactly what we want to control via the default and [benchmark] versions.

Thank you!

[PGijsbers]

I understand the concern for bloating the default installation with dependencies that aren't necessary. We can use this for now and clearly denote that it is using a benchmark configuration. That said, have you considered that it's also possible to add the hyperparameter and when set to "benchmark" check whether or not the flaml[benchmark] dependencies are installed? That could allow users of flaml[benchmark] to still use the non-benchmark configuration without requiring a second environment, without dependency bloat for flaml (default) users.

[qingyun-wu]

I understand the concern for bloating the default installation with dependencies that aren't necessary. We can use this for now and clearly denote that it is using a benchmark configuration. That said, have you considered that it's also possible to add the hyperparameter and when set to "benchmark" check whether or not the flaml[benchmark] dependencies are installed? That could allow users of flaml[benchmark] to still use the non-benchmark configuration without requiring a second environment, without dependency bloat for flaml (default) users.

This is a good suggestion! I will add such a configurable option (probably within a week). Will need to update this PR once it is added. Thank you!

[PGijsbers]

Any updates? If not I will probably simply raise an error when a user wants to run regular flaml when the benchmark dependencies are already installed (the error will tell them to re-install flaml without benchmark dependencies, or manually remove those dependencies if they want to run flaml in regular mode).

[PGijsbers]

Added benchmark support in #528, but looking forward to an updated integration script that can use the dependencies as intended for the new release :)

[qingyun-wu]

Thanks, @PGijsbers, an issue has been created accordingly.

[PGijsbers]

Is there an ETA on v2.0 (and updated integration script) btw?

[qingyun-wu]

Is there an ETA on v2.0 (and updated integration script) btw?

Hi @PGijsbers Thank you for asking. We don't have an ETA for now but will let you know once we have one. The automl functionality is not affected much by the changes in v2.0. So if you plan to do experiments, please feel free to use the older version. Thank you!

[PGijsbers]

Thanks for letting us know, will do!