Question on different feature_extractor in CLIP

[hxynjk]

Hello! This is a wonderful work in DG. I'm a newer to DG scene. We have noted that the CLIP(ViT/B-16) without training showed great high performance on DG datasets (like PACS). And the DPLCLIP has shown even high gains than CLIP. In my mind, we will also achieve large gains while we use the pretrained CLIP(Resnet 50). But I have tried to employ the CLIP(Resnet 50) as the backbone, I find the performance of DPLCLIP even show poor performance than those works with just Resnet 50 (like ERM, Fishr, MLDG...). So can you give me some suggestions on this situation? Thanks sincerely! Hope to receive your reply please!

[kxgong]

It seems that I also met the same problem. I run DPLCLIP(VIT-B16) on DG using default configurations (i.e., hype-parameters like lr). The average accs are: 72.2 (VLCS), 95.12 (PACS), and 75.9 (Office-Home). These figures should be much higher according to the paper. Should I tune the lr or other parameters? Looking forward for any replies! Also, this a wonderful work in DG, thanks for the authors' effort.

Here are the training scripts I used (maybe they are helpful to identify the error I made in reproduction): CUDA_VISIBLE_DEVICES=4,5,6,7 python -m domainbed.scripts.train --data_dir /nvme/gongkaixiong/dataset --output_dir my/DPLCLIP/VLCS/env0 --algorithm DPLCLIP --dataset VLCS --hparams "{\"clip_backbone\": \"ViT-B/16\"}" --test_envs 0

[zhyhan]

I have also encountered some difficulties in reproducing your results on the VLCS dataset. In Table 1 of the paper, the paper reported that DPLCLIP achieved an accuracy of 0.843, but when I ran the code, I only got 0.748. I did not change any hyperparameters or settings. Could you please help me figure out what went wrong?

[kxgong]

I have also encountered some difficulties in reproducing your results on the VLCS dataset. In Table 1 of the paper, the paper reported that DPLCLIP achieved an accuracy of 0.843, but when I ran the code, I only got 0.748. I did not change any hyperparameters or settings. Could you please help me figure out what went wrong?

The 0.843 performance is mostly like to appear in early steps/epochs. You can check your log again.

[zhyhan]

I attempt to choose the best performance (in early steps/epochs) to compute the average accuracy, and the performance is about 0.778 in the VLCS dataset.

I have also encountered some difficulties in reproducing your results on the VLCS dataset. In Table 1 of the paper, the paper reported that DPLCLIP achieved an accuracy of 0.843, but when I ran the code, I only got 0.748. I did not change any hyperparameters or settings. Could you please help me figure out what went wrong?

The 0.843 performance is mostly like to appear in early steps/epochs. You can check your log again.

[zhyhan]

I attempt to choose the best performance (in early steps/epochs) to compute the average accuracy, and the performance is about 0.778 in the VLCS dataset.

I have also encountered some difficulties in reproducing your results on the VLCS dataset. In Table 1 of the paper, the paper reported that DPLCLIP achieved an accuracy of 0.843, but when I ran the code, I only got 0.748. I did not change any hyperparameters or settings. Could you please help me figure out what went wrong?

The 0.843 performance is mostly like to appear in early steps/epochs. You can check your log again.

[kxgong]

I attempt to choose the best performance (in early steps/epochs) to compute the average accuracy, and the performance is about 0.778 in the VLCS dataset.

I have also encountered some difficulties in reproducing your results on the VLCS dataset. In Table 1 of the paper, the paper reported that DPLCLIP achieved an accuracy of 0.843, but when I ran the code, I only got 0.748. I did not change any hyperparameters or settings. Could you please help me figure out what went wrong?

The 0.843 performance is mostly like to appear in early steps/epochs. You can check your log again.

Do another run or carefully tune the learning rate. 😂