Merge with original transformers library

[salimmj]

🚀 Feature request

Merge this into the original transformers library.

Motivation

This library is awesome so thanks a lot but it would be much more convenient to have this merged into the original transformers library. The Huggingface team seems to be focused on adding lightweight options for their models and adapters are huge time-and-memory-savers for multitask use cases and would be a great addition to the transformers library.

Your contribution

You've done the integration here already so it should be straightforward but happy to help. I've posted an issue on huggingface's end as well.

[calpt]

Hi @salimmj,

Ultimately, merging our fork back into the original Transformers library is our goal. That would make adapters even more convenient to use, so we're definitely working on that! On the way, there are still some things we want to get right before making the integration step:

• add adapter support to more models available in Transformers: BERT & Co. are supported, DistilBERT almost ready (#67), GPT-2 requested and planned (#62), maybe with more to come
• add TF support: Currently, we only support PyTorch. Since Transformers also supports Tensorflow, we would want to port adapters to TF too
• new features still to come: There's a lot of research on adapters still going on in our Lab. Once published, we'll integrate new things into this library, probably with some breaking changes in the near future.

Since our resources are quite limited, I can't give a timeline when all this will be finished, but we're working on the integration!

[amirali1985]

Is there anything you would like contributors to do that might help accelerate this?

[calpt]

Hi @amirali1985, we're always happy to get help by contributors and any additional contributor definitely helps accelerating development on this project, so you're very welcome to contribute.

Here's what we're doing right now: Most of the changes currently are happening on the v2 branch, where we prepare our next major update with many new features (PR with technical description of changes). All new contributions should happen there.

Most importantly, we always need help supporting new transformer architectures. We currently have support for encoding models (BERT, RoBERTa, DistilBERT, XLM-RoBERTa) and first seq2seq models (GPT-2 and (m)BART), so there are still many more models which could be supported (ALBERT, XLM, Electra, Pegasus, T5 to just name a few interesting ones). If you're interested in adding adapters to a new architecture, there's a new detailed guide for the process. I'm also always happy to help with anything if you want some guidance to get started or have some additional questions.

[AmirAktify]

Thanks for the suggestions, @calpt. I work with T5 a lot, so I can take a stab at it.

[calpt]

@AmirAktify that would be awesome. I'll open an issue for T5 and assign you, so no one else works on that. I also just merged BART into the v2 branch a few minutes ago, which should be fairly similar. Let me know if you if there's anything I can help with.

[tanmaylaud]

I can help with ALBERT or Big Bird

[calpt]

@tanmaylaud Either would be great to have. Our dev branch is upgraded to the latest transformers version, so it already has Big Bird available: https://github.com/Adapter-Hub/adapter-transformers/pull/105. Please open an issue and assign yourself if you start working on one model. I'm happy to help if you have any questions or issues with the implementation.

[cody-moveworks]

Hi! I was wondering if there are updates with regards to merging this library back into transformers library?

[JoPfeiff]

We have been in close contact with people from Hugging Face, and have been heavily discussing options of either merging the two libraries, or disentangling the adapters from transformers. However, merging of the two libraries is quite difficult as there are too many moving parts on both sides, and disentangling is not possible without massive changes on the transformers side, while considerably hurting readability of adapters. So unfortunately the consensus (at least at the moment) is, that we are planning to pursue to have the two libraries separate from each other 😕.

[adapter-hub-bert]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[adapter-hub-bert]

This issue was closed because it was stale for 14 days without any activity.