SentenceMatchDecoder.py gives a wrong answer

[BluesBai]

I trained my model and got a 82% accuracy on dev set. Then I tried to evaluated it on test set, but it gave me 37% accuracy. I thought it was very strange, so I also tried to run SentenceMatchDecoder.py on my dev set, it again gave a 36% accuracy, which is not reasonable because it should be 82%. Then I found the correct_tags in SentenceMatchTrainer.evaluate() actually claculated the number of pairs which originally have label '1' but not the correct # of predict. I tried to go through your code in SentenceMatchGraph.py but still have no idea of what's wrong. Can you help me to figure it out?

[zhiguowang]

I didn't meet this issue before. Please try out the new code. Let me know if you still have the issue.

[zhengwsh]

I have the same issue. The accuracy given by SentenceMatchDecoder.py is strange, compared with the accuracy trained. Have it been fixed?

[netrookiecn]

I also have the same issue, can u help us check this question

[hexiahui]

Hi, I meet the same issue. While trying, I got a 83% acc on dev set. However, when I run SentenceMatchDecoder.py. The dev set only got 65.6% acc.

[hexiahui]

I found the problem. Just set fix_word_vec = True. If not, u should save the embeddings during ur training.