MemN2N-pytorch
MemN2N-pytorch copied to clipboard
nmhkahn
Implementation of End-to-End Memory Network in PyTorch
MemN2N-pytorch
PyTorch implementation of End-To-End Memory Network. This code is heavily based on memn2n by domluna.
Dataset
cd bAbI
wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz
tar xzvf ./tasks_1-20_v1-2.tar.gz
Training
python memn2n/train.py --task=3 --cuda
Results (single-task only)
In all experiments, hyperparameters follow the settings in memn2n/train.py (e.g. lr=0.001).
And since I suspect training is really unstable, I train the model 100 times in each task with fixed hyperparameters described in memn2n/train.py, then average top-5 results.
| Task | Training Acc. | Test Acc. | Pass |
|---|---|---|---|
| 1 | 1.00 | 1.00 | O |
| 2 | 0.98 | 0.84 | |
| 3 | 1.00 | 0.49 | |
| 4 | 1.00 | 0.99 | O |
| 5 | 1.00 | 0.94 | |
| 6 | 1.00 | 0.93 | |
| 7 | 0.96 | 0.95 | O |
| 8 | 0.97 | 0.89 | |
| 9 | 1.00 | 0.91 | |
| 10 | 1.00 | 0.87 | |
| 11 | 1.00 | 0.98 | O |
| 12 | 1.00 | 1.00 | O |
| 13 | 0.97 | 0.94 | |
| 14 | 1.00 | 1.00 | O |
| 15 | 1.00 | 1.00 | O |
| 16 | 0.81 | 0.47 | |
| 17 | 0.75 | 0.53 | |
| 18 | 0.97 | 0.92 | |
| 19 | 0.39 | 0.17 | |
| 20 | 1.00 | 1.00 | O |
| mean | 0.94 | 0.84 |
Issues
- It seems like model training heavily rely on weight initialization (or training is very unstable). For example, best performance of task 2 is ~90% however average performance over 100 experiments is ~40% with same model and same hyperparameters.
- WHY?
TODO
- Multi-task learning
nmhkahn