orion icon indicating copy to clipboard operation
orion copied to clipboard

add hpobench wrapper

Open donglinjy opened this issue 1 year ago • 2 comments

Description

Add an Orion benchmark task wrapper over HPOBench.

Tests

  • [x] I added corresponding tests for bug fixes and new features. If possible, the tests fail without the changes
  • [x] All new and existing tests are passing ($ tox -e py38; replace 38 by your Python version if necessary)

Documentation

  • [x] I have updated the relevant documentation related to my changes

Quality

  • [x] I have read the CONTRIBUTING doc
  • [x] My commits messages follow this format
  • [x] My code follows the style guidelines ($ tox -e lint)

donglinjy avatar Aug 30 '22 06:08 donglinjy

This could be a potential way to re-use singularity container in HPOBench, basically create it first when create Orion benchmark task and save its socket id, then create it again with this socket_id when run the task.

But this may not work in multiple worker scenario as we could create Orion benchmark task at local client node and run the task at a different node. https://github.com/automl/HPOBench/blob/d8b45b1eca9a61c63fe79cdfbe509f77d3f5c779/hpobench/container/client_abstract_benchmark.py#L106

donglinjy avatar Sep 21 '22 14:09 donglinjy

@bouthilx I already added the work-around for runner test case to succeed in https://github.com/Epistimio/orion/pull/993/commits/3afad9a78454cc5b7fb02d63887baccbe61db314. But there build will fail in python 3.10 test added recently, and HPOBench can only support to run up to python version 3.10.0. Any suggestion about this?

ERROR: Package 'hpobench' requires a different Python: 3.10.8 not in '<=3.10,>=3.6'

donglinjy avatar Dec 15 '22 06:12 donglinjy