ConvNext

[SimJeg]

Hello,

Have you considered using the ConvNext architecture for training DINOv2?

ConvNext has shown to have improved performance and lower latency on tasks such as CLIP. For example, in the open_clip repository, ConvNext-L@320 achieves better results with a +1.4% increase in zero-shot accuracy and is more than twice as fast as ViT-L@336.

While ViT may be easier to use in a "tokenize everything for my transformer world" approach, it's worth considering that CNNs still deserve... attention ^^

Best, Simon

[patricklabatut]

No, we have not (yet), but we might consider alternative architectures for the distilled models.

[patricklabatut]

Closing to keep track of similar asks in #166 instead