MWPToolkit icon indicating copy to clipboard operation
MWPToolkit copied to clipboard

Published 20 hours ago • verified publisher icon LYH-YF

Experiments for MAWPS-s

Open allanj opened this issue 2 years ago • 5 comments

Is the experiment for MAWPS-s using 5-fold as well? It seems yes to me as the paper reported. I got around 85.4 accuracy on MAWPS using train/dev/test. Wondering if I'm correct here. image

allanj avatar Nov 09 '21 01:11 allanj

yes, mawps-s is 5-fold setting.

LYH-YF avatar Nov 09 '21 02:11 LYH-YF

Thanks. Am I right that, for SVAMP, you are just directly doing train and test following the SVAMP paper?

allanj avatar Nov 09 '21 17:11 allanj

SVAMP is just a dataset for test, according to SVAMP paper, trainset consists of mawps and asdiv-a. And the setting is train-test split.running it with k-fold cross validation may not a good idea.

LYH-YF avatar Nov 10 '21 07:11 LYH-YF

Got it. Maybe should specify them in the table/paper?

From the table, it seems only those marked with "*" are train-test split.

allanj avatar Nov 10 '21 12:11 allanj

In the SVAMP paper, the appendix A show that the transformers with Roberta encoder obtain 38.9 accuracy image

But it seems the RobertaGen only get 30.3 here. Curious about the difference here

allanj avatar Nov 12 '21 10:11 allanj