Try Gomez-Bombarelli's TerminalGRU

[pechersky]

I was looking to see whether the paper's authors had any draft code for the model they describe, to check whether they did something differently. I found that R GB had implemented some new RNN layers they might have used for the decoder. I don't know if they ended up using the conventional GRu or this new TerminalGRU instead. Might be worth trying out:

https://github.com/fchollet/keras/compare/master...rgbombarelli:master#diff-3118e4e28157032506f771f279a551c3R639

[maxhodak]

Hey I'm finally back in CA and have fixed my DL machine. This is an interesting idea... I'll get to this in the next couple days!

[hsiaoyi0504]

Maybe this is what we expect fchollet/keras/#694 and fchollet/keras/#3947.

[rgbombarelli]

That layer is quite helpful - each neuron gets to see not only the probability distribution, but also what character got sampled. In a perverse case, say you have 51-49 and 49-51 chances of opening a bracket "(" in two successive characters, without teacher forcing, you end up with "((" 25% of the time.

It's essentially teacher forcing (I didn't know it was called that when I wrote the TerminalGRU code. Here is a modern way to do it. seq2seq also has many useful tools around RNN's and might be a good place to start from. We also released the VAE from the paper here in case it's useful.

[hsiaoyi0504]

Wow, @rgbombarelli Is it the official release of implementation of your original paper?