Advanced hyperparameter configuration

[david-thrower]

1) I recommend:

To talos.Scan(), add a parameter: "hyperparameter_config": for example, a dictionary in the format:

KEY: 'name of hyperparameter as listed in talos.Scan(params)' ;

VALUE: list of dictionaries, one dict for each config option for the hyperparameter named in the key, to add additional options like what np.choice offers (e.g size argument allowing an array of selections from a given param, and an option for selection [with | w/o] replacement).

Example:

params =\
    {'l4_upstream_conn_index':
        np.arange(1,3).tolist(),
     'l5_upstream_conn_index':
        np.arange(1,4).tolist(),
     'l6_upstream_conn_index':
        np.arange(1,5)).tolist(),
     'l7_upstream_conn_index':
        np.arange(1,6).tolist()}

param_options = {

    'l4_upstream_conn_index':[
        {'size':3},
        { 'with_replacement':1}],  # Select 3 elements with replacement from l4_upstream_conn_index 

    'l5_upstream_conn_index':[
        {'size':4},
        { 'with_replacement':1}] , # Select 4 elements with replacement from l5_upstream_conn_index
# ...
}

def make_skip_connection_model(params)

    layers = np.ones(7).tolist()
    
    layers[0] = Input((5,))

    layers[1] = Dense(7)(layers[0] ) 
    layers[2] = Dense(7)(layers[1] ) 
    layers[3] = Dense(7)(layers[2] ) 

    for i in np.arange(4,8):
        layers[i] =\
            Dense(
                Concatenate([layers[c]) 
                                          for c in params[f'l{i}_upstream_conn_index'']],
                                           axis=1)
            )  # To make the skip connections to myltiple predecessor layers, this needs to make multiple selections from this parameter...

        out_layer = Dense(1,'sigmoid')(layers[-1])
        model = Model(inputs=layers[0],
                                 outputs=out_layer)

        model.compile(...)
        results = model.fit(...)
        
        return results, model 
talos.scan(model=make_skip_connection_model,
                 params = params,
                 hyperparameter_config = param_options)

[mikkokotila]

From the example, is not completely clear why would you not do this same just by adding the variety you want in the parameter dictionary i.e. why does it have to be outside of it the way things are implemented currently?