Various initializers and batch normalization

An implementation of weight/bias initializers and batch normalization in Tensorflow.

MNIST database is used to show performance-comparison

Network architecture

In order to examine the effect of initializers and batch normalization, a simple network architecture called multilayer perceptrons (MLP) is employed.

MLP has following architecture.

• input layer : 784 nodes (MNIST images size)
• first hidden layer : 256 nodes
• second hidden layer : 256 nodes
• output layer : 10 nodes (number of class for MNIST)

Various initializers

The following initializers for weights/biases of network are considered.

• Weight Initializer
  • normal, trucated normal
  • xaiver : Understanding the difficulty of training deep feedforward neural networks
  • he : Delving Deep into Rectifiers
• Bias Initializer
  • normal, constant

Simulation results

• Weight Initializer : he > trauncated normal = xaiver > normal
• Bias Initilaizer : zero > normal

Sample results are following.

Batch normalization

Batch normalization improves performance of network in terms of final accuracy and convergence rate. In this simulation, bias initalizer was zero-constant initializer and weight initializers were xavier / he.

Sample results are following.

Usage

python run_main.py --weight-init <weight initializer> --bias-init <bias initializer> --batch-norm <True or False>

<weight initializer> must be selected in [normal, truncated_normal, xavier, he]. <bias initializer> must be selected in [normal, zero].

You may command like python run_main.py --weight-init xavier --bias-init zero --batch-norm True.

Acknowledgement

This implementation has been tested on Tensorflow r0.12.