Clarification on hidden_layers_sizes

[dareneiri]

My first question: If I use the following code below, is deepsurv indeed running without any hidden layers?:

hyperparams = {
    'n_in': 25,
    'learning_rate': 0.0001,
    'hidden_layers_sizes': None, 
    'lr_decay': 0,
    'momentum': 0.9,
    'L2_reg': 0.0,
    'L1_reg': 0,
    'activation': 'rectify',
    'dropout': 0,
    'batch_norm': False,
    'standardize': False,
}
network = deepsurv.DeepSurv(**hyperparams)

I can train the model: python log = network.train(train_data, valid_data, n_epochs = 1300) This results in a good concordance value. I just want to verify what is being passed. What were your predefined ranges you used for Optunity?

My second question is when I specify hidden_layers_sizes: When I run the model with hyper-parameters as above, but with hidden_layers_sizes specified,

hyperparams = {
   ...
    'hidden_layers_sizes': [14,2], 
   ...
    'activation': 'rectify',
   ...
}

Then I get the error:

/home/spark/anaconda2/envs/theano-fips/lib/python2.7/site-packages/lasagne/layers/dense.pyc in get_output_for(self, input, **kwargs)
    122         if self.b is not None:
    123             activation = activation + self.b
--> 124         return self.nonlinearity(activation)
    125 
    126 

TypeError: 'str' object is not callable

It's only when I comment out the activation does it model actually attempt to train. But the code should regardless be running rectify, correct?

[baharian]

I believe the default value for activation is lasagne.nonlinearities.rectify:

In[]:  ?deepsurv.DeepSurv
Init signature: deepsurv.DeepSurv(self, n_in, learning_rate, hidden_layers_sizes=None, lr_decay=0.0, momentum=0.9, L2_reg=0.0, L1_reg=0.0, activation=<function rectify at 0x10da18488>, dropout=None, batch_norm=False, standardize=False)
Docstring:      <no docstring>
Init docstring:
This class implements and trains a DeepSurv model.
Parameters:
    n_in: number of input nodes.
    learning_rate: learning rate for training.
    lr_decay: coefficient for Power learning rate decay.
    L2_reg: coefficient for L2 weight decay regularization. Used to help prevent the model from overfitting.
    L1_reg: coefficient for L1 weight decay regularization
    momentum: coefficient for momentum. Can be 0 or None to disable.
    hidden_layer_sizes: a list of integers to determine the size of each hidden layer.
    activation: a lasagne activation class.
        Default: lasagne.nonlinearities.rectify
    batch_norm: True or False. Include batch normalization layers.
    dropout: if not None or 0, the percentage of dropout to include after each hidden layer. Default: None
    standardize: True or False. Include standardization layer after input layer.
File:           //anaconda/lib/python2.7/site-packages/deepsurv/deep_surv.py
Type:           classobj

[dareneiri]

@baharian Thanks. I think I confused myself because I was using the pypi version of deepsurv (updated last year) yet was reviewing the code on the master branch here (updated just a few months ago). The default activation changed from lasagne.nonlinearities.rectify to "rectify"