BERT-of-Theseus icon indicating copy to clipboard operation
BERT-of-Theseus copied to clipboard

Published 20 hours ago • verified publisher icon JetRunner

CoLA reproducibility

Open mcps5601 opened this issue 3 years ago • 2 comments

Hi, I cannot reproduce the CoLA score as same as the one on paper. I followed the HuggingFace's repo to train a predecessor model with Matthew correlation score of 55.76. However, the hightest score of the successor model I got is 35.82. Could you provide the hyperparameter set for training on the CoLA dataset?

mcps5601 avatar Dec 17 '21 13:12 mcps5601

Hi @mcps5601, have you solved the issue? I train a predecessor model with Matthew correlation of 58.04 on the dev set but could not reproduce the result of successor. The highest score I get is about 44. I try to tune the learning_rate and scheduler_linear_k but it looks like they do not refine the results too much. CC @JetRunner

mengdiz97 avatar Jul 26 '22 18:07 mengdiz97

Hi @mcps5601, have you solved the issue? I train a predecessor model with Matthew correlation of 58.04 on the dev set but could not reproduce the result of successor. The highest score I get is about 44. I try to tune the learning_rate and scheduler_linear_k but it looks like they do not refine the results too much. CC @JetRunner

May I ask how your parameters are configured

whynot-zyl avatar Dec 31 '23 10:12 whynot-zyl