Inconsistent of reproduced results and paper's results.

[dercaft]

Hi,

I tried to reproduce the results for VTAB given in paper, but I have the following results:

The results are pretty different to given ones.

And here is my cifar_100 result:

Here is my running environment:

Hardware: 3090*4

Python 3.10.0

CUDA&Pytorch:

1. cuda 11.6.1
2. pytorch 1.13.1
3. torchvision 0.14.1
4. timm 0.6.5

I use a newer version of toolkit because of some compatible problem happened between 3090 driver and cuda 10, cudnn 7

[ZhuWenjie98]

Hi, did you manage to implement the fine tune results with these codes, I found there is a huge gap between the paper and this codebase's.

[dercaft]

Hi, did you manage to implement the fine tune results with these codes, I found there is a huge gap between the paper and this codebase's.

I only change the environment to newer version of Pytorch. I have this GAP problem too.

[ross-Hr]

This code is completely different from the idea in the paper, for example, there is no freezing of parameter layers during training(i found the ANS in https://github.com/dongzelian/SSF/issues/3#issuecomment-1310287258).

And the code did not implement reparameterization technology during the inference phase.

I am very confused.

[csguoh]

I also try to reproduce the result, and the result of my reproduced result on cifar100 is also 93.81, instead of the reported 93.99. looking forward to the authors' reply.

[dongzelian]

@csguoh I have uploaded the log (https://github.com/dongzelian/SSF/blob/main/log/cifar100.csv) of cifar100 for your reference. The gap between 93.81 and 93.99 is not large. You can try to change the number of GPUs to 8 with batch size 16 of each GPU.

[csguoh]

I will try it again. Thanks for your advice!

[Allencheng97]

For VTAB-1K, since its setting is very close to few shot which is highly depend on the random seed, my personal suggestion is fixed the cuda,pytorch,torch.backends seed, this also may cause the difference. It seems the author code based on the timm repo. Timm only fix the pytorch seed but other random factors are not fixed. I don't know the author. But I am actually quite surprised someone questioned the 0.18 acc difference...