Pluggable Model Integration Interface

[calpt]

This PR drafts a new model integration interface which makes it easier to support new and custom model architectures for selected adapter methods without full model implementation.

This is done with the new AdapterModelInterface class that translates from generic model access points to model-specific attribute names.

Example usage:

For Qwen model:

model_interface = AdapterModelInterface(
    adapter_types=["lora", "reft"],
    model_embeddings="embed_tokens",
    model_layers="layers",
    layer_self_attn="self_attn",
    layer_cross_attn=None,
    attn_k_proj="k_proj",
    attn_q_proj="q_proj",
    attn_v_proj="v_proj",
    layer_intermediate_proj="mlp.up_proj",
    layer_output_proj="mlp.down_proj",
)

model_name = "Qwen/Qwen2-0.5B"

model = AutoModelForCausalLM.from_pretrained(model_name)
adapters.init(model, interface=model_interface)

config = LoRAConfig()
# config = LoReftConfig()
model.add_adapter("my_adapter", config=config)

print(model.adapter_summary())

Overview

Supported adapter types:

• [x] LoRA
• [x] ReFT
• [ ] Bottleneck/ Compacter
• [ ] Prefix Tuning
• [ ] Prompt Tuning

Supported features:

• [ ] Embedding training
• [ ] Parallel composition
• [ ] Fusion composition