adapters
adapters copied to clipboard
Add (IA)^3 implementation
This PR adds the (IA)^3 adapter architecture proposed in "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" by Liu et al. (2022).
- Paper: https://arxiv.org/pdf/2205.05638.pdf
- Reference implementation: https://github.com/r-three/t-few
- Implementation in adapter-transformers:
-
IA3Configclass that inherits fromLoRAConfig - New config options
attn_matrices,composition_modeandno_decompositionto support scaled composition of (IA)^3
-