orion
orion copied to clipboard
add hpobench wrapper
Description
Add an Orion benchmark task wrapper over HPOBench.
Tests
- [x] I added corresponding tests for bug fixes and new features. If possible, the tests fail without the changes
- [x] All new and existing tests are passing (
$ tox -e py38
; replace38
by your Python version if necessary)
Documentation
- [x] I have updated the relevant documentation related to my changes
Quality
- [x] I have read the CONTRIBUTING doc
- [x] My commits messages follow this format
- [x] My code follows the style guidelines (
$ tox -e lint
)
This could be a potential way to re-use singularity container in HPOBench
, basically create it first when create Orion benchmark task and save its socket id, then create it again with this socket_id when run the task.
But this may not work in multiple worker scenario as we could create Orion benchmark task at local client node and run the task at a different node. https://github.com/automl/HPOBench/blob/d8b45b1eca9a61c63fe79cdfbe509f77d3f5c779/hpobench/container/client_abstract_benchmark.py#L106
@bouthilx I already added the work-around for runner test case to succeed in https://github.com/Epistimio/orion/pull/993/commits/3afad9a78454cc5b7fb02d63887baccbe61db314. But there build will fail in python 3.10 test added recently, and HPOBench
can only support to run up to python version 3.10.0. Any suggestion about this?
ERROR: Package 'hpobench' requires a different Python: 3.10.8 not in '<=3.10,>=3.6'