horde-ad icon indicating copy to clipboard operation
horde-ad copied to clipboard

Implement LSTM with both soft and hard attention

Open Mikolaj opened this issue 2 years ago • 2 comments

Implements a couple of variants of LSTM. Then let's create some example neural networks with that (e.g., modify the existing MNIST RNN to use LSTM). If not and if no libraries offer a good alternative, let's implement our own. Two good blog posts about LSTM: https://jasdeep06.github.io/posts/Understanding-LSTM-in-Tensorflow-MNIST and https://colah.github.io/posts/2015-08-Understanding-LSTMs.

This should be fun and not too hard, but understanding and perhaps tweaking our MNIST RNN first could help.

See #41 and https://github.com/Mikolaj/mostly-harmless/discussions/16?sort=new#discussioncomment-2811053. In particular, let's implement both soft and hard attention, if possible, and see if that trains.

Mikolaj avatar May 24 '22 12:05 Mikolaj