Generalized ia3

[IanMagnusson]

What's Here

Moves a more generalized IA3 adaptor implementation to Tango (PR pending) and provides an example script for how to use it in Catwalk.

Results on piqa

While hardly impressive results, the IA3 implementation manages to reduce validation loss and recover much of the accuracy of the fully tuned equivalent for all the architectures for which default configurations are provided. The gpt-j-6b full tune is not able to run on a single gpu while the IA3 training is able to fit due to having far fewer optimizer states for its fewer trainable parameters.