adapters icon indicating copy to clipboard operation
adapters copied to clipboard

Add (IA)^3 implementation

Open calpt opened this issue 3 years ago • 0 comments

This PR adds the (IA)^3 adapter architecture proposed in "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" by Liu et al. (2022).

  • Paper: https://arxiv.org/pdf/2205.05638.pdf
  • Reference implementation: https://github.com/r-three/t-few
  • Implementation in adapter-transformers:
    • IA3Config class that inherits from LoRAConfig
    • New config options attn_matrices, composition_mode and no_decomposition to support scaled composition of (IA)^3

calpt avatar Jul 25 '22 16:07 calpt