adapters icon indicating copy to clipboard operation
adapters copied to clipboard

Add support for AdapterPlus

Open julian-fong opened this issue 4 months ago • 0 comments

This PR aims to add support for AdapterPlus

Github: https://github.com/visinf/adapter_plus Paper: https://arxiv.org/pdf/2406.06820

Integration of AdapterPlus into the adapters library will involve adding new parameters/options to the BnConfig

Checklist of things that are added/to be added

  1. New type of scaling called channel, in which we add learnable parameters for the channel/input_size dimension
  2. New type of init_weights called houlsby, where the projection matrices $W_{down}$ and $W_{up}$ will be initialized with zero-centered Gaussian with a standard deviation of $10^{-2}$ truncated at 2 standard deviations, and zeros for bias
  3. Support for drop_path, also known as stochastic depth ONLY applicable for vision based tasks

julian-fong avatar Oct 19 '24 01:10 julian-fong