Bad performance on ImageNet variants

[tjddus9597]

I ran the FLYP code to compare with "Masked Images Are Counterfactual Samples for Robust Fine-tuning, CVPR 2023", using ViT-B/32 model. I expect that FLYP can be competitive with other methods, but the performance on OOD datasets of model trained with FLYP is significantly degraded.

Zero-shot CLIP performance using ViT-B/32 is the following: ImageNet Top-1 accuracy: 63.4
ImageNetV2 Top-1 accuracy: 55.9
ImageNetR Top-1 accuracy: 69.3 ImageNetSketch Top-1 accuracy: 42.3 ImageNetA Top-1 accuracy: 31.4

I ran just one epoch training with FLYP, but its performance is: ImageNet Top-1 accuracy: 73.3
ImageNetV2 Top-1 accuracy: 62.6
ImageNetR Top-1 accuracy: 63.1 ImageNetSketch Top-1 accuracy: 40.9 ImageNetA Top-1 accuracy: 25.9

FLYP cannot preserve the robustness, and the performances on ImageNet-R, ImageNet Sketch, and ImageNet-A are dropped compared to Zero-shot CLIP, even just trained for an epoch. I use the same parameters that are used in training for ViT-B/16 experiments.

Can you clarify this phenomenon? Are there any wrong things in this experiment?