RBM Weight Updates Issue

[Darwin2011]

Such Issues are in your RBM different implementation. [python version] self.W += lr * (numpy.dot(self.input.T, ph_sample) - numpy.dot(nv_samples.T, nh_means)) [C++ version] W[i][j] += lr * (ph_mean[i] * input[j] - nh_means[i] * nv_samples[j]) / N;

For there two versions, the weight update methods are incosistent. And Actually I think the right version should be self.W += lr * (numpy.dot(self.input.T, ph_means) - numpy.dot(nv_means.T, nh_means)) Could you please help me confirm such issues? I am not quite sure that whether it's issue or not. And I am just a freshman for deep learning.

Best Regards

[liulhdarks]

@Darwin2011 I think (ph_mean[i] * input[j] - nh_means[i] * nv_samples[j]) is right. The weights gradient should be data-model。

[Darwin2011]

@liulhdarks ,thanks The following code shows that the different weight update method. Could you check the following link? https://github.com/echen/restricted-boltzmann-machines/blob/master/rbm.py

[liulhdarks]

@Darwin2011 https://github.com/echen/restricted-boltzmann-machines/blob/master/rbm.py doesn't use the visible layer after sampling when updating weights. You can refer to the follow ML frameworks.

• [Deeplearning4j] https://github.com/agibsonccc/java-deeplearning/blob/master/deeplearning4j-core/src/main/java/org/deeplearning4j/models/featuredetectors/rbm/RBM.java

INDArray wGradient = input.transpose().mmul(probHidden.getSecond()).sub( nvSamples.transpose().mmul(nhMeans));

• [darks-learning] https://github.com/liulhdarks/darks-learning/blob/master/src/main/java/darks/learning/neuron/rbm/RBM.java

DoubleMatrix wGradient = input.transpose().mmul(hProp1.prob) .sub(vPropn.sample.transpose().mmul(hPropn.prob));

Hope to help you!

[Darwin2011]

@liulhdarks thanks for your kind help. I will follow your guide.

Best Regards

[luochao436]

@liulhdarks @yusugomori I think the update for W , a , b should as follows: self.W += lr * (numpy.dot(self.input.T, ph_mean - numpy.dot(nv_samples.T, nh_means)) self.vbias += lr * numpy.mean(self.input - nv_samples, axis=0) self.hbias += lr * numpy.mean(ph_mean - nh_means, axis=0)

In origin python RBM.py. , self.W and self.hbias is not correct. This problem is also in C/C++ files, but the W has been fixed. If I am not right, please inform me. Thanks.