horde-ad
horde-ad copied to clipboard
Implement LSTM with both soft and hard attention
Implements a couple of variants of LSTM. Then let's create some example neural networks with that (e.g., modify the existing MNIST RNN to use LSTM). If not and if no libraries offer a good alternative, let's implement our own. Two good blog posts about LSTM: https://jasdeep06.github.io/posts/Understanding-LSTM-in-Tensorflow-MNIST and https://colah.github.io/posts/2015-08-Understanding-LSTMs.
This should be fun and not too hard, but understanding and perhaps tweaking our MNIST RNN first could help.
See #41 and https://github.com/Mikolaj/mostly-harmless/discussions/16?sort=new#discussioncomment-2811053. In particular, let's implement both soft and hard attention, if possible, and see if that trains.