cruise
cruise copied to clipboard
Introduce various weight initialization methods
In the current DNN implementation, the weights of layers such as fully connected layers and convolutional layers are initialized with random values from a Gaussian distribution. It is known that learning deep neural network is largely influenced by initial weight values as well as neural network architecture. Thus, we should introduce various methods to initialize weight values. For example, Caffe has "weight filler" which fills initial weight parameters of layers. There are many types of weight fillers such as Constant, Gaussian, Positive Unit ball, Uniform, Xavier, MSRA and Bilinear. We also introduce these methods to get better training performance of deep neural networks.