why can't I replicate some results in serveral datasets ？

[pierowu]

I've tried to replicate the results in LoRA setting by using PEViT/scripts/lora_clip.sh in solid intact. My device is a V100 GPU. The code env is torch 1.7.0 and CUDA 11.0. The results i get are close to the ones in the article. However, in 4 datasets, the results seem to deviate significantly from those in the article. Do you have any insight of this problem?

[jkooy]

Hi, thanks again for interests. What seeds are you using for these results? I just checked, my results are:

For FER2013, the acc for seed 0,1,2 are 50.878, 50.655, and 51.574, on average 51.04 KITT is 35.302, 54.008, 54.852, on average 48.05 MNIST is 65.070, 58.250, 60.750, on average 61.36 Flowers is 82.089, 74.492, 66.247, on average 74.28

Currently I don't have a V100 GPU to test. But from my experience the hardware difference can indeed lead to slightly different results. Appended my logs obtained on A6000 for all these results https://drive.google.com/file/d/1hl4sX3-KzMrx3LUD3ynh5k3gXtCovvN0/view?usp=share_link FYI. It is normal here to have variance because of the few-shot setting.

[pierowu]

Thank you for your carefully reply. Here is my results under different seed. It seems that the results will variate greatly sometimes according to different seeds. Even if I choose the same seeds, the results still seem to variate in different devices, which brings barriers for fair comparison.

[jkooy]

Yes. It is normal as we do 5-shot training and the samples are different for each seed. That's why we took the average for three seeds across all 20 datasets for mitigating the variance. For devices, I would suggest developing methods under a same env when using the toolkit so that the findings are consistent.

On Wed, Oct 18, 2023 at 12:27 AM pierowu @.***> wrote:

Thank you for your carefully reply. Here is my results under different seed. [image: image] https://user-images.githubusercontent.com/61963313/276141004-55de3c3c-a62b-46bb-91aa-886ae363b931.png It seems that the results will variate greatly sometimes according to different seeds. Even if I choose the same seeds, the results still seem to variate in different devices, which brings barriers for fair comparison.

— Reply to this email directly, view it on GitHub https://github.com/eric-ai-lab/PEViT/issues/8#issuecomment-1767832834, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZ4OCV3L4UIZYJCEHZAMLTX76AHXAVCNFSM6AAAAAA6EYSAYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRXHAZTEOBTGQ . You are receiving this because you commented.Message ID: @.***>