Train adapters without Hugging Face Trainer scripts

[bhavitvyamalik]

Hi, I was looking into example scripts for Adapter-Hub and almost all *_no_trainer.py scripts were not using adapters at all. Are you guys planning to add those scripts soon? I can also help in porting trainer scripts to no_trainer scripts if someone can guide me about what all changes will be required for that. Thank you!

cc: @calpt

[calpt]

Hey @bhavitvyamalik, sorry for the late answer.

It's true that all scripts without the trainer class haven't been modified to support adapter training. However, the modification should be fairly straightforward as the main modification to be made is the model setup (we have a description for these changes in our documentation). Otherwise, training scripts should be re-usable without any major modifications (maybe some minor things like only checkpointing adapters and not full models etc.). One can also have a look at the modifications made to the trainer scripts which mostly involve these lines of code.

Hope this helps!

[mistel1225]

Hi, I have a question about train an adapter without the trainer class, In fine-tuneing a model, we usually call model.train() method to enable all parameters to be trainable, However, in adapter case, we need to freeze all parameters except for adapter and also downstream header, so should i still call model.train() in adapter case or should we replace it by model.train_adapter()?

[dv-fenix]

Hey @calpt,

I'd love to work on this. While the modifications are pretty straightforward, I think that it would save inexperienced users some time if the code were to be made available here.

I was thinking of modifying the run_mlm_no_trainer.py script to support adapter training. This can then be used as a template to modify the other *_no_trainer.py scripts according to their specific use cases. Do you mind if I open a PR to incorporate these changes?

[calpt]

Hey @dv-fenix, sorry for totally missing your comment on this issue. We're of course always happy for any contributions towards making the code more accessible for users and your suggestion sounds great. Please go ahead if you're still interested :)

[dv-fenix]

Hey @calpt, I will take this up then! Will send in a PR soon 😄

[adapter-hub-bert]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[adapter-hub-bert]

This issue was closed because it was stale for 14 days without any activity.

[vatsal-kr]

Hey @dv-fenix Can you share the run_mlm_no_trainer.py code updated for adapters if you've made any progress?