ipynb is down

[cosmozhang]

The ipynb was not fixed..

[JonathanRaiman]

Thanks! Good catch. Just fixed a typo in the ipynb.

[cosmozhang]

Hi Jonathan,

 I am trying to implement dropout in the theano.scan, but without 

using your package, since I have written most part of the rnn. Do you have any suggestions? I read your codes and instructions, but still have some questions. E.g, since masks is non_sequences, what will be the input in the theano.function? Thanks very much!

Cosmo

On 04/20/2015 10:18 PM, Jonathan Raiman wrote:

Thanks! Good catch. Just fixed a typo in the ipynb.

— Reply to this email directly or view it on GitHub https://github.com/JonathanRaiman/theano_lstm/issues/11#issuecomment-94615352.

Best, Cosmo Zhang

[JonathanRaiman]

Hey Cosmo,

To get dropout to work I've done the following:

Get some random number generator:

import theano, theano.tensor as T
srng = theano.tensor.shared_randomstreams.RandomStreams(1234)

Then create the mask you want to use:

drop_prob = T.scalar()
shape = (300, 300)
mask = T.cast(
    srng.binomial(n=1, p=1-drop_prob, size=shape),
    theano.config.floatX
)

And then in your scan:

result, updates = theano.scan(fn = step,
                     sequences = [x],
                     outputs_info = etc...,
                     non_sequences = [mask])

error = etc...

Your theano function can now take as inputs:

func = theano.function([x, target, drop_prob], (result[-1] - target)**2)

As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.

[cosmozhang]

Thank you very much, I solved it last night in another way:

import theano, theano.tensoras T srng= theano.tensor.shared_randomstreams.RandomStreams(1234)

drop_prob= T.scalar() shape= (shape1, 300,300) #shape1 is the length of input sequence mask= T.cast( srng.binomial(n=1,p=1-drop_prob,size=shape), theano.config.floatX )

and then

|result, updates = theano.scan(fn = step, sequences = [x, mask], outputs_info = etc..., non_sequences = None)|

Do you think it is also plausible?

And in your setting, how did yo write step function?

Thank you very much!

On 4/21/2015 10:47 PM, Jonathan Raiman wrote:

Hey Cosmo,

To get dropout to work I've done the following:

Get some random number generator:

import theano, theano.tensoras T srng= theano.tensor.shared_randomstreams.RandomStreams(1234)

Then create the mask you want to use:

drop_prob= T.scalar() shape= (300,300) mask= T.cast( srng.binomial(n=1,p=1-drop_prob,size=shape), theano.config.floatX )

And then in your scan:

|result, updates = theano.scan(fn = step, sequences = [x], outputs_info = etc..., non_sequences = [mask])

error = etc... |

Your theano function can now take as inputs:

|func = theano.function([x, target, drop_prob], (result[-1] - target)**2) |

As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.

— Reply to this email directly or view it on GitHub https://github.com/JonathanRaiman/theano_lstm/issues/11#issuecomment-95006889.