I can't produce the result in this paper

[hertz-pj]

I configured the same environment and ran the run_augmentation_sst5_low.sh file and got 15 identical results. The result is "Final Test Acc: 28.6425". Is there some error in the code?

[tanyuqian]

Thanks for your feedback. We checked our code and it works well on our side.

What's your environment setting? e.g., versions of python, PyTorch and pytorch_pretrained_bert. The code magic_module.py operates some low-level variables in nn.Module, so please keep these versions as strict as possible:

python3.6.5 torch==1.0.1 pytorch_pretrained_bert==0.6.1

[hertz-pj]

I try again on the environment as same as yours. python3.6.5 torch==1.0.1 pytorch_pretrained_bert==0.6.1 It not work, the Final result is 28.6425 again..

[marscrazy]

I suppose you see "Final Test Acc: 28.6425" is the last run of the 15 experiments. Technically, there are 14 more "Final Test Acc". I got avg:36.71 and std:3.63.

[tanyuqian]

When we install a wrong version of pytorch, we get the same situation as you said (the number is also 28.6425).

Could you double check your environment? like delete related files in your .cache/ and /tmp/.