[FR] define hyperparameters outside @hyperopt macro

[KronosTheLate]

I currently have quite a few parameters I am optimizing for. Additionally, I want to run several @hyperopt runs to get some statistics on the quality of the output. My loop has the form

hohb = @hyperopt for resources=B,
                sampler=Hyperband(R=η^s_max; η, inner=BOHB(dims=[Continuous(), Categorical(3), Categorical(3), 
                Continuous(), Categorical(6), Continuous(), Continuous(), Continuous(), Categorical(3), 
                Continuous(), Categorical(6), Categorical(3), Continuous()])),
                dropoutrate = 0:0.01:0.5,
                optimizer = [Descent, Momentum, ADAM],
                first_layer = [BatchNorm(chin), BatchNorm(chin, relu), identity],
                n_layers_convblock1 = 2:5,
                activatation_convblock1 = [tanh, σ, relu, relu6, leakyrelu, rrelu],
                kernel_convblock1 = [(2, 2), (3, 3), (4, 4)],
                stride_convblock1 = [2, 3, 4, 5],
                chout = (2 .^ (1:6)),
                conv_to_dense_function = [GlobalMaxPool(), GlobalMeanPool(), identity],
                n_layers_denseblock1 = 1:3,
                activatation_denseblock1 = [tanh, σ, relu, relu6, leakyrelu, rrelu],
                loss_and_final_layer = [(Flux.Losses.logitcrossentropy, identity), (Flux.Losses.crossentropy, softmax), (Flux.Losses.focal_loss, softmax)],
                learningrate_multiplyer = range(0.1, 10, 100)

, and within the body I have

push!(configs, ID=>(;dropoutrate, optimizer, first_layer, n_layers_convblock1, activatation_convblock1, 
                        kernel_convblock1, stride_convblock1, chout, conv_to_dense_function, n_layers_denseblock1, 
                        activatation_denseblock1, loss_and_final_layer, learningrate_multiplyer))

, where ID gets defined if no state is defined.

This is however not super flexible and quite noisy.

Would it be possible to define e.g. a tuple or vector of Pair{Symbol, some_collection} that could be passed to the @hyperopt macro? Something like

my_hyperparams = (:resources=B,
:sampler=>Hyperband(R=η^s_max; η, inner=BOHB(dims=[Continuous(), Categorical(3), Categorical(3), 
Continuous(), Categorical(6), Continuous(), Continuous(), Continuous(), Categorical(3), 
Continuous(), Categorical(6), Categorical(3), Continuous()])),
:dropoutrate => 0:0.01:0.5,
:optimizer => [Descent, Momentum, ADAM],
:first_layer => [BatchNorm(chin), BatchNorm(chin, relu), identity],
:n_layers_convblock1 => 2:5,
:activatation_convblock1 => [tanh, σ, relu, relu6, leakyrelu, rrelu],
:kernel_convblock1 => [(2, 2), (3, 3), (4, 4)],
:stride_convblock1 => [2, 3, 4, 5],
:chout => (2 .^ (1:6)),
:conv_to_dense_function => [GlobalMaxPool(), GlobalMeanPool(), identity],
:n_layers_denseblock1 => 1:3,
:activatation_denseblock1 => [tanh, σ, relu, relu6, leakyrelu, rrelu],
:loss_and_final_layer => [(Flux.Losses.logitcrossentropy, identity), (Flux.Losses.crossentropy, softmax), (Flux.Losses.focal_loss, softmax)],
:learningrate_multiplyer => range(0.1, 10, 100))

The ability to define the parameter names and spaces outside the loop would simplify my code.

It would also be awesome to be able to define the hyperparameter-name, hyperparameter-space and Categorical/Continuous together - the current solution of defining these things on seperate lines in seperate formats is tedious for a large number of hyperparameters. Automating the counting of the number of categorical values could also be done. Perhaps something like

test_params = [(:dropoutrate, 0:0.1:0.5, Continuous), (:optimizer, [Descent, Momentum, ADAM], Categorical)]
function make_dims_for_BOHB(params)
    @assert only(unique(length.(params)))==3 "All elements in the input vector are expected to have length 3"
    @assert false ∉ [param[3] <: Hyperopt.LatinHypercubeSampling.LHCDimension for param in params]
    return [param[3]==Categorical ? param[3](length(param[2])) : param[3]() for param in params]
end

[baggepinnen]

It looks like it works to define arrays of candidate values outside the macro expression:

using Hyperopt, Optim, Random
f(a;c=10) = sum(@. 100 + (a-3)^2 + (c-100)^2)

avec = LinRange(1,5,1800)
cvec = exp10.(LinRange(-1,3,1800))

hohb = @hyperopt for resources=50, sampler=Hyperband(R=50, η=3, inner=RandomSampler()), a = avec, c = cvec
    if !(state === nothing)
        a,c = state
    end
    @show state
    res = Optim.optimize(x->f(x[1],c=x[2]), [a,c], SimulatedAnnealing(), Optim.Options(f_calls_limit=round(Int, resources)))
    Optim.minimum(res), Optim.minimizer(res)
end
plot(hohb)

defining both the key and the value outside is probably not supported at the moment, but you could perhaps use the manual iteration interface. If you'd feel like implementing new features, I'd be up to discuss it, but I unfortunately have rather little time available to adding new features to this package at the moment.

[KronosTheLate]

Then I will do the more manual and tedious explicit writing out of stuff for now. Completely fair that your time is limited, and thank you for the package.

I will leave this issue open as a quality-of-life feature request ^_^