ENH: Orthogonal LoRA layer initialization (2)

[BenjaminBossan]

Continuation of, and supersedes #2389

Check discussion there for further info.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

I compared the results with orthogonal init vs normal and Gaussian LoRA init, with all other parameters kept equal, using the MetaMathQA method comparison suite. Test accuracy on GSMK8K is 47.8% for default, 49.3% for Gaussian, and 48.9% for orthogonal. As expected, memory usage and runtime are practically identical.

I also plotted the train loss:

There seems to be a slight advantage for orthogonal initialization there is well, though for the first half of the run, it lags behind the other methods.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

ping @githubnemo