MLOS icon indicating copy to clipboard operation
MLOS copied to clipboard

Latest version of FLAML changes optimization behavior

Open bpkroth opened this issue 1 year ago • 0 comments
trafficstars

Recently a new version (v2.2.0) of FLAML was released and it introduces some new warnings and behavior:

INFO     flaml.tune.searcher.blendsearch:blendsearch.py:169 No low-cost partial config given to the search algorithm. For cost-frugal search, consider providing low-cost values for cost-related hps via 'low_cost_partial_config'. More info can be found at https://microsoft.github.io/FLAML/docs/FAQ#about-low_cost_partial_config-in-tune
WARNING  flaml.tune.searcher.suggestion:suggestion.py:555 You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
WARNING  flaml.tune.searcher.suggestion:suggestion.py:555 You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
FAILED mlos_bench/mlos_bench/tests/optimizers/toy_optimization_loop_test.py::test_flaml_optimization_loop - assert 63.77786073141286 == 60.15 ± 6.0e-01
  
  comparison failed
  Obtained: 63.77786073141286
  Expected: 60.15 ± 6.0e-01

We should investigate and fix these.

For now, I'm creating a PR to simply pin the version to avoid them until we have time to do that.

bpkroth avatar Aug 12 '24 21:08 bpkroth