PEFT implementations of adapters are outdated and languishing

[bghira]

System Info

Latest PEFT, transformers, diffusers codebase.

Who can help?

@benjaminbossan @sayakpaul

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder
• [ ] My own task or dataset (give details below)

Reproduction

The LoKr, LoHa, and other LyCORIS modules are all outdated and missing substantial improvements and fixes from their upstream implementations.

Expected behavior

Either the upstream implementation should be directly wrapped, or these implementations should be kept up to date by the PEFT team. It doesn't seem like it should be on KohakuBlueleaf to update support for his adapters since the code was hastily copy-pasted into PEFT before being abandoned.

[BenjaminBossan]

We're currently in a bit dev starved when it comes PEFT (basically just me), which will hopefully be resolved soon. Could you give an example of what is broken or missing in LoKr and LoHa? Wrapping probably won't be an option, as the implementations differ too much and we have different requirements in PEFT.

[bghira]

full matrix tuning, 1x1 convolutions, quantised LoHa/LoKr, weight-decomposed LoHa/LoKr, fixed rank dropout implementation, fixed maths (not multiplying against the vector, but only the scalar)

[bghira]

basically the results are entirely different from upstream to PEFT implementation, with the upstream author recommending peft be avoided at this point in time :[

[BenjaminBossan]

Thanks for the info. I reformatted this and put what sounds like bugs to the top, as they are more important. If you have any further references, like GH issues, that would be fantastic.

• [ ] fixed rank dropout implementation
• [ ] fixed maths (not multiplying against the vector, but only the scalar)
• [ ] full matrix tuning
• [ ] 1x1 convolutions
• [ ] quantised LoHa/LoKr
• [ ] weight-decomposed LoHa/LoKr

[sayakpaul]

Thanks for making us aware of this, @bghira!

@BenjaminBossan WDYT about also opening this to the community in case someone is interested in collaborating to start with some of these?

[BenjaminBossan]

Yes, absolutely @sayakpaul. Do you have a suggestion how to best promote this?

[sayakpaul]

@BenjaminBossan https://github.com/huggingface/peft/issues/1935

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[bghira]

no stalebot. bad

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.