RSM binarization not being used

[alexminnaar]

In the following code in def rsm_learn

                # If probabilities are higher than randomly generated, the states are 1
                randoms = rand.rand(batch_size, self.num_hid)
                pos_hid = array(randoms < pos_hid_prob, dtype=int)

                # Negative phase - generate data from hidden to visible units and then again to hidden units.
                neg_vis = pos_vis
                neg_hid_prob = pos_hid
                for i in range(self.gibbs_steps):  # There is only 1 step of contrastive divergence
                    neg_vis, neg_hid_prob, D, p = self.__contrastive_divergence_rsm__(neg_vis, pos_hid_prob, D)
                    if i == 0:
                        perplexity += p

It doesn't not look like pos_hid is actually being used. It looks like it is being assigned to neg_hid_prob which is then immediately overwritten by the contrastive divergence output. Or is pos_hid being used somewhere else?

Perhaps pos_hid should have been passed to __contrastive_divergence_rsm__ rather than pos_hid_prob such that the visible layer is being reconstructed from the binarized hidden layer rather than the probabilities?

[larsmaaloee]

Hi again,

The reason for this is that we are using the positive hidden probabilities for CD instead of the states. Please read https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf to see why that is. It is a design choice. The reason why both are present in the code is for transparency.

Best regards

---

Lars Maaløe PHD Student Cognitive Systems, DTU Compute Technical University of Denmark (DTU)

Email: [email protected], [email protected] Phone: 0045 2229 1010 Skype: lars.maaloe LinkedIn http://dk.linkedin.com/in/larsmaaloe DTU Orbit http://orbit.dtu.dk/en/persons/lars-maaloee(0ba00555-e860-4036-9d7b-01ec1d76f96d).html

On 01 Oct 2015, at 22:17, Alex Minnaar [email protected] wrote:

In the following code in def rsm_learn

If probabilities are higher than randomly generated, the states are 1

            randoms = rand.rand(batch_size, self.num_hid)
            pos_hid = array(randoms < pos_hid_prob, dtype=int)

            # Negative phase - generate data from hidden to visible units and then again to hidden units.
            neg_vis = pos_vis
            neg_hid_prob = pos_hid
            for i in range(self.gibbs_steps):  # There is only 1 step of contrastive divergence
                neg_vis, neg_hid_prob, D, p = self.__contrastive_divergence_rsm__(neg_vis, pos_hid_prob, D)
                if i == 0:
                    perplexity += p

It doesn't not look like pos_hid is actually being used. It looks like it is being assigned to neg_hid_prob which is then immediately overwritten by the contrastive divergence output. Or is pos_hid being used somewhere else?

Perhaps pos_hid should have been passed to contrastive_divergence_rsm rather than pos_hid_prob such that the visible layer is being reconstructed from the binarized hidden layer?

— Reply to this email directly or view it on GitHub https://github.com/larsmaaloee/deep-belief-nets-for-topic-modeling/issues/5.