segmentation_models.pytorch icon indicating copy to clipboard operation
segmentation_models.pytorch copied to clipboard
verified publisher icon qubvel-org

VIT Adapter

Open aselimc opened this issue 3 months ago • 4 comments

Hello,

I propose adding ViT adapter option in TimmUniversalEncoder for ViTs that does not have multi-scale features, so that SMP can now support a whole new set of backbones. The following links can be used as a reference/starting point.

DINOv3_Adapter DINOv2_Adapter Original VIT Adapter Repo AdaptersHub Adapters HF Version

Or perhaps these type of adapters might actually better be on timm? @rwightman @qubvel

aselimc avatar Sep 21 '25 09:09 aselimc

+1

ogencoglu avatar Sep 29 '25 20:09 ogencoglu

Hi @aselimc @ogencoglu 👋

I’d like to take this up and work on adding ViT adapter support in TimmUniversalEncoder. Before I begin, could you please confirm the intended integration point: should the adapter layer live within the TimmUniversalEncoder itself (as an internal submodule), or be attached externally similar to how the DINOv2_Adapter repo structures it?

Thanks for opening this.

ha405 avatar Oct 05 '25 11:10 ha405

Any preferences @qubvel ?

ogencoglu avatar Oct 05 '25 12:10 ogencoglu

I agree this is a design decision in which project owner @qubvel should comment on it.

aselimc avatar Oct 05 '25 13:10 aselimc