Regarding adapter heads

[yeoserene]

Hi! I'm new to the whole adapters training regime, so pardon me for asking ignorant questions.

I trained two types of adapters on an extractive question answering task - 1 on SQUAD2.0 and 1 on another dataset. All of the data are formatted according to the SQUADv2.0 format published on datasets repository, and they both train to succession, following closely the run_qa.py script provided. When trying to now use them for AdapterFusion, I ran the following setups and found out the following problems:

1. This piece of code could run and train to completion, but I think the heads are missing? So performance is real bad.

model=AutoModelForQuestionAnswering(...)
model.load_adapter('adapter1', with_head=False)
model.load_adapter('adapter2', with_head=False)
adapter_setup = Fuse('adapter1', 'adapter2')
model.add_adapter_fusion(adapter_setup)
model.set_active_adapters(adapter_setup) 
model.train_adapter_fusion(adapter_setup)

2. So I thought, I'll add a QA head. Unfortunately, even though I kept the rest of the script the same, it calls for a KeyError: [loss] error, which suggests that the model somehow couldn't compute the loss for optimisation. I would think this error probably comes from the fact that the inputs required using this add_qa_head function is different from the AutoModelForQuestionAnswering used in Trainer? Or it could very well be me not understanding the right things...

model=AutoAdapterModel(...)
model.load_adapter('adapter1', with_head=False)
model.load_adapter('adapter2', with_head=False)
adapter_setup = Fuse('adapter1', 'adapter2')
model.add_adapter_fusion(adapter_setup)
model.set_active_adapters(adapter_setup) 
model.add_qa_head("head_name")
model.train_adapter_fusion(adapter_setup)

3. Lastly, I tried the following which is to set one of the adapter head to be true, and it is still running. But I honestly don't know what I am doing now LOL.

model=AutoModelForQuestionAnswering(...)
model.load_adapter('adapter1', with_head=False)
model.load_adapter('adapter2', with_head=**True**)
adapter_setup = Fuse('adapter1', 'adapter2')
model.add_adapter_fusion(adapter_setup)
model.set_active_adapters(adapter_setup) 
model.train_adapter_fusion(adapter_setup)

I would really appreciate some help here. Thank you so much!

[github-actions[bot]]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[adapter-hub-bert]

This issue was closed because it was stale for 14 days without any activity.