adapters
adapters copied to clipboard
Add support for AdapterPlus
This PR aims to add support for AdapterPlus
Github: https://github.com/visinf/adapter_plus Paper: https://arxiv.org/pdf/2406.06820
Integration of AdapterPlus into the adapters
library will involve adding new parameters/options to the BnConfig
Checklist of things that are added/to be added
- New type of scaling called
channel
, in which we add learnable parameters for the channel/input_size dimension - New type of
init_weights
calledhoulsby
, where the projection matrices $W_{down}$ and $W_{up}$ will be initialized with zero-centered Gaussian with a standard deviation of $10^{-2}$ truncated at 2 standard deviations, and zeros for bias - Support for
drop_path
, also known as stochastic depth ONLY applicable for vision based tasks