interpret_bert icon indicating copy to clipboard operation
interpret_bert copied to clipboard

Published 20 hours ago • verified publisher icon ganeshjawahar

WC task bert accuracy is much lower than paper claimed

Open JizeCao opened this issue 5 years ago • 1 comments

I have tried use the SentEval WC dataset to evaluate Bert performance. But the result is much lower than the paper claimed. (0.4 compares to 24.9 in paper for layer 0). I achieve similar performance on all other 9 tasks.

JizeCao avatar Jan 03 '20 07:01 JizeCao

It's about one month and still no one reply ... The issue is relatively critical because I have tried different language models with current codebase and none of them has a reasonable result on this one. Is there any reason behind that? I am very grateful if someone can reply me.

JizeCao avatar Feb 03 '20 23:02 JizeCao