Low accuracy

[dujiaxin]

This looks like a wonderful model. However, when I run your model according to your README.md, it gave me em,f1,prec,recall all about 0.04. Is there any thing I should change besides default settings?

[anushkmittal]

@dujiaxin were you able to figure it out?

[cdmalon]

Same problem:

{'label': 'ema', 'score': {'em': 0.041188386225523295, 'f1': 0.04091829844699527, 'prec': 0.04091829844699527, 'recall': 0.04091829844699527, 'doc_em': 0.0, 'doc_f1': 0.0, 'doc_prec': 0.0, 'doc_recall': 0.0, 'sp_em': 0.0, 'sp_f1': 0.0, 'sp_prec': 0.0, 'sp_recall': 0.0, 'joint_em': 0.0, 'joint_f1': 0.0, 'joint_prec': 0.0, 'joint_recall': 0.0}}

[nixiaoyu]

I met the same problem. Have you solved it?

{'label': 'ema', 'score': {'em': 0.04010803511141121, 'f1': 0.04010803511141121, 'prec': 0.04010803511141121, 'recall': 0.04010803511141121, 'doc_em': 0.0, 'doc_f1': 0.0, 'doc_prec': 0.0, 'doc_recall': 0.0, 'sp_em': 0.0, 'sp_f1': 0.0, 'sp_prec': 0.0, 'sp_recall': 0.0, 'joint_em': 0.0, 'joint_f1': 0.0, 'joint_prec': 0.0, 'joint_recall': 0.0}}

[easonnie]

You can try to change the seed in the following code snippet: https://github.com/easonnie/semanticRetrievalMRS/blob/149833a7ca9146ef4a89d90e0b6cedd0ab088e72/src/qa_models/hotpot_bert_v0.py#L130