Cannot re-produce the result on SNLI

[ZacharyChenpk]

I have downloaded the pre-trained model from https://drive.google.com/open?id=1Yn-WCw1RaMxbDDNZRnoJCIGxMSAOu20_ and https://s3-us-west-2.amazonaws.com/allennlp/models/srl-model-2018.05.25.tar.gz accessed here, and tried to re-produce the experiment on SNLI dataset with the following environment settings: python 3.6 allennlp 0.8.1 torch 1.8.0+cu111

When the evaluation finished, I found that the test accuracy is only 0.8563 (= 8412 / 9824) and dev acc is 0.8557, which are far lower than the results reported in the paper. Though the different module versions might lead to performance decreasing, is the drop of about 0.06 reasonable?

[ZacharyChenpk]

Or is there any available trained model for higher pytorch/cuda version? My GPU is with higher driver version but small memory, which makes it impossible to use torch 1.0.0, and hard to train a new model from scratch. Maybe using a model trained on correct pytorch/cuda version can solve it?

[cooelf]

Sorry for a late reply. This might be because of the version mismatch as the repo has continuously updated after the model release.

Could you replace the L1024 with L1024 in the modeling.py file and try again?

https://github.com/cooelf/SemBERT/blob/master/pytorch_pretrained_bert/modeling.py#L1024

https://github.com/cooelf/SemBERT/blob/master/pytorch_pretrained_bert/modeling.py#L1023