[Feature Request] Add callbacks option

[jakubczakon]

First off, great job on the library!

Secondly, I think it would be a good idea to have an option to define callbacks. For example, I would like to log the progress live to Neptune tracking tool.

I made this example experiment public so you can go there if you want. Full script is available here.

Anyhow, what I ended up doing was:

class NeptuneLogger:
    def new_config(self, *args, **kwargs):
        pass
    
    def __call__(self, job):
        neptune.send_metric('run_score', job.result['loss'])
        neptune.send_text('run_parameters', str(job.kwargs['config']))

...

optim = BOHB(configspace = worker.get_configspace(),
                 run_id = RUN_ID,
                 nameserver=ns_host, 
                 nameserver_port=ns_port,
                 result_logger=NeptuneLogger())

It gets the job done but having to define this new_config method is weird. Also, if I had more callbacks that I wanted to do, it would be tricky.

I think a good place to either have a separate callbacks argument that accepts a list of callbacks (callables on the job) or have an option to register them on an initiated optimizer:

def neptune_callback(job):
     neptune.send_metric('run_score', job.result['loss'])
     neptune.send_text('run_parameters', str(job.kwargs['config']))
...
optim.register_callbacks([neptune_callback])

What do you think?

[sfalkner]

Thanks for trying out HpBandSter; I'm glad you like it! As you have already successfully figured out, the result_logger is kind of a callback that is notified every time a new configuration is proposed, or a new result comes in. IMHO, the 6 lines of code you had to write to achieve that are not too bad. I agree that having callbacks would be nice, but it's actually not that hard to add more things to your logger to perform exactly the actions you need.

So far, you are the first one I know that actually wrote a custom logger. How would you feel about adding your custom logger to the examples for others to see how to do that?

[jakubczakon]

I guess you are right @sfalkner. Callbacks are a "nice to have", but definitely you can do pretty much whatever you need in the __call__ function.

I would love to add an example of that! I will do that next week as I am on vacation right now.

PS I am writing a blog series about HPO in python and HpBandSter is going to be part 5 of that. Part 0 and Part 1 are already published on towards data science.

[jakubczakon]

Hi @sfalkner. I have been on and off trying to add this example but I am having trouble understanding how to build it. When I worked with rtd and sphinx in my projects I would create .rtd or .ipynb files and then run make html to create html files. In this project, it seems a bit different and it would help me a lot if you could give a simple step-by-step instruction of how to build it. Specifically:

• where should I create a new .ipynb with the example?
• which .rt d or .html should I modify to change the index.html for Examples-How to use HpBandSter ?
• how to build it? I'd presume once the first and second questions are answer I just need to run make html

Thanks!

[sfalkner]

Hi @jakubczakon, sorry for the delay on my end. IIRC, you just put the example as a .py file into the hpbandster/examples' folder. Make sure it has example` in its name, so it is picked up during building the documentation. This line makes sure that the examples are automatically picked up without changing anything else. Let me know if you have any problems. Thanks for putting in the time!

[jakubczakon]

Thanks @sfalkner ! Added example as you proposed in PR #65