model.load_head() and save.adapter() DOUBTS

[Ch-rode]

Environment info

• adapter-transformers version: @ git+https://github.com/adapter-hub/adapter-transformers.git@021d1a67a1dee58d80e68b782c0dcbd9de0f8ea9
• Platform:
• Python version: 3.9.7
• PyTorch version (GPU?): torch==1.9.0+cu111

Details

Hello, I have trained an adapter based on my data using a BertAdapterModel as pretrained one. Unfortunately, I have not clear from the documentation how to deal with the classification head. I wanted to try both options:

1. add a classification head and save it within the adapter
2. add a classification head but save it as separated

This is my model:

model.add_adapter("NUM1adapter",set_active=True)
model.add_classification_head("NUM1adapter_head", num_labels=2, layers=2,activation_function='tanh')
model.active_head == 'NUM1adapter_head'
model.train_adapter(["NUM1adapter"])

1. OPTION1: adapter + classification head

After the training I have: model.save_adapter(path,with_head=True) Then I load it and if i print the model I can see the head in the structure named NUM1adapter_head model.load_adapter(best_model_path,with_head=True) model.set_active_adapters(['NUM1adapter'])

but when I call the model it doesn't recognize my head: logits = model(b_input_ids, b_attn_mask).logits AttributeError: 'BaseModelOutputWithPoolingAndCrossAttentions' object has no attribute 'logits'

1. OPTION2: adapter and classification head separated

After the training I have: model.save_adapter(path,'NUM1adapter',with_head=False) model.save_head(path,'NUM1adapter_head')

When I load them, if I do: model.load_adapter(best_model_path) model.load_head(best_model_path) model.set_active_adapters(['NUM1adapter'])

everything is fine during my training, but I have the warning that the head is overwrite.

• Why I cannot include the head in the save_adapter and load it directly?
• And how I can avoid the warning (head overwrite) in the second option?
• Have I to do something to fine-tuning also the classification head or model.active_head == 'NUM1adapter_head' is enough ?

Thank you very much for the help!

[calpt]

Hey @Ch-rode,

Why I cannot include the head in the save_adapter and load it directly?

This only works automatically if the prediction head has exactly the same name as the adapter. So, your description of Option 1 should work if you rename your head to "NUM1adapter"

And how I can avoid the warning (head overwrite) in the second option?

This warning occurs because load_adapter() will also load a prediction head from the given model path, therefore already loading the "NUM1adapter_head" together with the adapter in the line model.load_adapter(best_model_path). When you load the head a second time using model.load_head(best_model_path), it will overwrite the initially loaded head. To make sure no head is loaded with load_adapter(), you can pass with_head=False (equivalently to the save_adapter() method).

Have I to do something to fine-tuning also the classification head or model.active_head == 'NUM1adapter_head' is enough ?

The head is fine-tuned whenever it is activated, so yes, setting active_head is enough.

Hope this helps!

[adapter-hub-bert]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[adapter-hub-bert]

This issue was closed because it was stale for 14 days without any activity.