missing `model.embeddings` in the `RobertaAdapterModel`

[Wangt-CN]

• adapter-transformers version: 3.1.0a0
• Platform: Ubuntu20
• Python version: 3.7
• PyTorch version (GPU?): 1.12.0 GPU

Details

Hi, thanks for this great repo and I am a fresher. I found that the default Roberta model from transformer contains the embeddings module model.embeddings by using RobertaModel.from_pretrained('roberta-base').

However, by loading the adapter-transformers model, such embeddings module is lost and I am very confused. The command:

model = RobertaAdapterModel.from_pretrained("roberta-base")
model.add_adapter('test', config=HoulsbyConfig())
model.train_adapter('test')

The error log:

AttributeError: 'RobertaAdapterModel' object has no attribute 'embeddings

Because in my understanding, this embeddings model should be loaded in adapter model by default? Thanks a lot for your help.

[Wangt-CN]

I have just found that the embeddings module is in RobertaAdapterModel.roberta.

Therefore, a simple method is to just set model = RobertaAdapterModel.roberta.

What's the difference of RobertaAdapterModel and RobertaAdapterModel.roberta? Can I directly initialize my model with RobertaAdapterModel.roberta?

[calpt]

Hey, I'll try to clarify this a bit:

RobertaModel and RobertaAdapterModel are used for different application cases. RobertaModel provides the base Transformer architecture, including embeddings and Transformer layers, but without any task-specific layers (prediction heads) at the output of the model. To use a RoBERTa model for a downstream task, HuggingFace Transformers provides different classes for different types of tasks, e.g. RobertaForSequenceClassification for classification tasks. These classes are built on top of the base RobertaModel class which, together with layers and embeddings, is usually accessible via e.g. RobertaForSequenceClassification.roberta.

RobertaAdapterModel is similar to these task-specific classes in a sense. It also contains a base RoBERTa model in model.roberta. However, RobertaAdapterModel provides more flexibility as it allows you to add multiple different prediction heads for different tasks, e.g. add_classification_head() for sequence classification (docs). As embeddings are always provided by the base model, they are accessible via model.roberta.embeddings and not via model.embeddings.

Hope this helps!