Can't reproduce BLEU scores listed in README

[jeffquach]

Hey,

I downloaded your notebook and pretrained weights and wasn't able to reproduce the BLEU scores you obtained. These are the BLEU scores I got: [('100-1', 2.2396683846707894e-155), ('1000-1', 2.031826294518176e-155)]

I also tried training the network myself to see if I'd get a different result but received similar BLEU scores to the ones above.

I haven't modified your code.

Are there certain considerations I may not have accounted for prior to running your code, or is it likely I did something wrong?

[mangoyuan]

Same error. I just tell what i meet. Every time i call build_vocab, the word2idx and idx2word will be different. So i save the vocab to pickle first time and train. When i test, i just load the pickle file to restore the vocab. Because the author use jupyter, the vocab is the same in a .ipython file. Another problem i observed is that repeating call sample method by the same network and input, the bleu score will rise. I am not familiar with NLP, so i don't why. Just like

[*] Load Best-1.ckpt
0.2985686199496656
[*] Load Last.ckpt
0.7313586221143726
[*] Load Best-1.ckpt
0.74649989186069
[*] Load Last.ckpt
0.7313586221143726

May this help you~