regularization on biases

[GuangQin1995]

if regularizer is not None: regularizers = sum([tf.nn.l2_loss(variable) for variable in self.variables]) loss += (regularizer * regularizers) it seems like that you have regularization on biases, as the self.variables included the biases ` variables = [] for w1,w2 in weights: variables.append(w1) variables.append(w2)

for b1,b2 in biases:
    variables.append(b1)
    variables.append(b2)`

[jakeret]

Yes you're right. Have you checked if it makes a big difference?

According to deeplearning.stanford.edu/ :Applying weight decay to the bias units usually makes only a small difference to the final network.

But I might be worth investigating

[meijie0401]

I think when batchsize is not 1, we should divide 'regularizers' by 2batch size just like following. What's your idea? regularizers = sum([tf.nn.l2_loss(variable) for variable in self.variables])//(2batch_size)

[jakeret]

Sorry for the very late reply. The power of the regularizer is a hyperparameter like the batch size we can cover their relation in the param search instead of having a it implicitly in the code, don't we?