MogrifierLSTM

MogrifierLSTM copied to clipboard

Does your code replicate the results in Mogrifier LSTM paper?

No. I attempted to implement it and test it in a basic RNN network for text prediction using the Brown corpus but didn't get very promising results compared to a regular LSTM. Perhaps with a larger dataset, more training time, better network architecture, or better hyperparameter tuning I could have come up with something more definitive. I believe my implementation is correct, but it's hard to verify without spending more time with the aforementioned criteria (this was a weekend project).

Looking at the paper, it seems they did plan to release the code (they even put a link that does not work: https://github.com/deepmind/lamb ), but something has been holding the writer up. I scoured the internet and found a couple comments (that I can't seem to find any more) between the author and another researcher that seemed to indicate the author was having trouble getting their official implementation working properly.

Thank you for your detailed reply! I've emailed the author.

Welcome! I'd be interested to hear if they tell you anything new.

@qlan3 thanks to #2 I seem to have something approaching the results of the paper. Would need more testing to fully verify, but it seems promising. Hope you find what you need!

Thanks!