Unable to load CLIP model in weaviate `flax-community/clip-rsicd-v2`

[samhodge-aiml]

This is the hugging face model we wish to work with in Weaviate

from transformers import AutoProcessor, AutoModel

processor = AutoProcessor.from_pretrained("flax-community/clip-rsicd-v2")

model = AutoModel.from_pretrained("flax-community/clip-rsicd-v2")

But when one of the Weaviate staff Marcin Antas tried to help us integrate it he reported

I actually tried to download your model that you want to run with our clip module and I get an error:
ValueError: Unrecognized configuration class <class 'transformers.models.clip.configuration_clip.CLIPConfig'> to build an AutoTokenizer.
Can you create an issue for that? I think we need to add support for your type of model to make it to work with our clip inference container

So I guess we need to think about how to support this model within the Weaviate framework.

Sorry I cannot give more details.

perhap @antas-marcin can elucidate some details for us.

Note the model was edited from v4 to v2 the v4 model is not functional