Convert to open_clip checkpoint

[code-ishwar]

Is it possible to convert the checkpoint to open_clip format?

I tried the below but doesn't work. Error is mentioned below and returns the same file.

model = VisionTextDualEncoderModel.from_pretrained(ckpt_path, from_flax=True)
model.save_pretrained("openclip-ckpt")

You are using a model of type hybrid-clip to instantiate a model of type vision-text-dual-encoder. This is not supported for all configurations of models and can yield errors. /home/miniforge-pypy3/lib/python3.10/site-packages/transformers/modeling_flax_pytorch_utils.py:460: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1711403388920/work/torch/csrc/utils/tensor_numpy.cpp:206.) pt_model_dict[flax_key] = torch.from_numpy(flax_tensor) All Flax model weights were used when initializing VisionTextDualEncoderModel.

All the weights of VisionTextDualEncoderModel were initialized from the Flax model. If your task is similar to the task the model of the checkpoint was trained on, you can already use VisionTextDualEncoderModel for predictions without further training.