Adding CLIP Pretrained Weights to ResNet

[Luciennnnnnn]

Requesting the addition of CLIP pretrained weights to ResNet model. This would greatly improve the performance of ResNet by allowing it to leverage the powerful representations learned by CLIP in downstream tasks. It would also make it easier for users to transfer knowledge between ResNet and CLIP without having to train their own models from scratch.

Thank you.

[rwightman]

@LuoXin-s The OpenAI CLIP ResNets have an attention pooling layer that makes the models quite a bit larger than same depth ResNet (significant FLOPs and params in the layer), and less flexible (no longer fully convolutional).

I do have an impl of this layer but wasn't clear it's worth going the next step to add support, the ConvNeXt CLIP models I trained are much better and remain fully convolutional....

[Luciennnnnnn]

@rwightman OpenAI's models are trained with their private dataset, and I found that their model tend capturing more information related to perceptual quality other than those trained by laion.