How to change CLIP to BiomedCLIP

[YangQiuEric]

Question

I am doing medical image project, so how can we change CLIP to other models?

[zsxm1998]

change the vision_tower parameter

[20191864218]

change the vision_tower parameter

change the vision_tower parameter

Hi, could you please be more specific? Which specific parts do you need to change?Thank you

[YangQiuEric]

@zsxm1998 @20191864218 I just change the vision_tower to microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224, and the finetune can be done. But the model_vqa.py has errors, can not recognize this model. I tried to download all the files from hugging face, but it does not have preprocess_config.json and config.json. How to fix?

[ATing0203]

@zsxm1998 @20191864218 I just change the vision_tower to microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224, and the finetune can be done. But the model_vqa.py has errors, can not recognize this model. I tried to download all the files from hugging face, but it does not have preprocess_config.json and config.json. How to fix?

Have you solved the problem yet? @YangQiuEric

[Abhiram-kandiyana]

@zsxm1998 @20191864218 I just change the vision_tower to microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224, and the finetune can be done. But the model_vqa.py has errors, can not recognize this model. I tried to download all the files from hugging face, but it does not have preprocess_config.json and config.json. How to fix?

Any update @YangQiuEric ?