DiffCSE icon indicating copy to clipboard operation
DiffCSE copied to clipboard
verified publisher icon voidism

Can not reproduce this issue

Open zyznull opened this issue 3 years ago • 3 comments

I have used this run_diffcse.sh, but the avg sts result is only 77.00. Is there some thing need to notice?

zyznull avatar Jun 30 '22 02:06 zyznull

Same here, got the result of 77.33.

Gitsamshi avatar Jul 26 '22 17:07 Gitsamshi

Me too.

hbin0701 avatar Aug 04 '22 05:08 hbin0701

Here also :( Below is my result, with batch size and etc. same as written in paper.

image

qmin2 avatar Aug 18 '22 04:08 qmin2

Hi all!

After I tried some experiments on another machine, I found that the hyperparams are very sensitive to the device you use. I cannot reproduce the results on another machine with the same hyperparams either. Your python/pytorch/cuda/huggingface version will affect your results. The hyperparams on the paper are only suitable for the first machine I used, so you probably need to re-search hyperparams to get the same results on your machine.

A recent CSE paper https://github.com/yiren-jian/NonLing-CSE/tree/main/VisualCSE seems also have this problem for CSE. They suggest use the same hardwares and software versions to faithfully reproduce the results in the paper.

voidism avatar Sep 25 '22 04:09 voidism