Performance of the AR model

[SimonOst]

trafficstars

Hi,

I tried to use the Stanford Attentive Reader on the data, testing the pretrained model viabash test_SAR.sh. I downloaded both the glove embeddings, which I put under data/embedding, and the RACE data, under data/data. However, the resulting accuracy on both development and test data is only around .36-.39, and not > .4, as stated in the paper. There is no error message as far as I can tell, the output of the script is appended. Any idea about what could have gone wrong?

Looking forward to an answer!

log.txt

[michaelpulsewidth]

Hi,

This is probably due to a vocabulary mismatch. We used nltk.sent_tokenize and nltk.word_tokenize as our tokenizers, which have some inherent randomness. Could you try to retrain the model using your parsed results? It should be able to achieve a reasonable performance though the numbers might not be exactly the same. To build a new vocabulary, you can comment line 211 and uncomment line 210 in the main.py, which are

Line 210: #word_dict = utils.build_dict(train_examples[0] + train_examples[1] + train_examples[2], args.max_vocab_size) Line 211: word_dict = pickle.load(open("../obj/dict.pkl", "rb"))

I heard from several people that they had reproduced the performance in PyTorch, so there should not be a significant problem.

[SimonOst]

Hello, thanks for the fast response! I am currently retraining the model and will let you know how it worked when it's finished.

[Tagbo-Aroh]

Hi, I'm new to python and am new to NLP. I'm trying to do a project with your dataset. I tried loading your model with codes below; with gzip.open(r'''C:\Users\arohi\Downloads\a4\model_GA.pkl.gz''', 'rb') as f: train_set = cPickle.load(f)

but am getting an error message; 'ascii' codec can't decode byte 0xd4 in position 0: ordinal not in range(128)