Feature request: Automatic hyperparamter optimization

[heborras]

Hi all,

a use case for which I've been using seml a lot is grid searches for finding good hyperparamters for neural nets. And it would be great if one could automate this process somewhat, since at the moment one has to go from a coarse grain search to a fine grained one in multiple dimensions at the same time. Which usually involves lots of looking at plots and running analysis scripts. It would be nice if I could instead basically tell seml: Please run a thousand experiments and find a good hyperparameter config within these parameter ranges. And with a good hyperparamter searcher the result would likely be better than what I could have come up with. I know seml is somewhat agnostic to what for a program one runs, not limited to neural nets, but fundamentally many published hyperparamter searchers also are agnostic in this way (though many do focus on NNs). So I think fitting algorithms can be found in literature.

So I was wondering: Do you think there is a way to extend seml in such a way that it could support such hyperparamter search algorithms? Or rather: Do you see this as a possible extension in the future? Or is this out of scope for seml?

Some general thoughts of mine on this are: One would obviously need to move away from the somewhat static principle of returning results only at the end of a run and one would probably also need to introduce a way to make seml be able to dynamically create/fetch hyperparamters as slots in slurm become available. These are likely somewhat drastic changes to the codebase, but some of the fundamentals already exist in sacred (even though if I understood the general sentiment correctly, then seml wants to move away from sacred in the long run).

A little bit more context: In the past I've been a big fan of what determined.ai does with their Adaptive ASHA searcher: https://docs.determined.ai/latest/model-dev-guide/hyperparameter/search-methods/hp-adaptive-asha.html But their system doesn't play nicely with slurm on a conceptual level. Probably what Weights and Biases does is conceptually more in-line with how seml works: https://docs.wandb.ai/tutorials/sweeps#-pick-a-method However, in general the field of automatic hyperparamter optomization has been a very active one and I think one of the most feature complete suite of searchers is Syne Tune, which is however tightly coupled to aws: https://github.com/awslabs/syne-tune Still, maybe one can use one of these as an initial spring-board to get started.

I'd be happy to put some dev-time into this in the next year. Since I am seeing many more hyperparamter searches with seml in my near future. That is however dependent on if there is enough interest for such a feature and if there is willingness to maintain it afterwards as well. So if this were to happen, then I think some good coordination would be required, so that people are on the same page.