When fine-tune with fix_vit=False, will the CLIP-VIT be tuned, or the PORA adapter be tuned?

[lrybbbccc]

I am tuning the 4KHD model for downstream tasks, and I have such question because after fine-tuning, the checkpoint directory does not have checkpoint for CLIP-VIT, and the build_mlp.py is rewritten as vision_tower = 'openai/clip-vit-large-patch14-336'