Benjamin Bossan

@bghira Do you happen to have a use case where you could test if this PR works and is working well enough speed-wise? I think the implementation could be ready...

> stupid question but is the code example above complete? i dont see the hinge loss function You mean the code right at the top? No, it's not complete at...

> Not sure if it's a problem with the lora_alpha parameter, since it works fine when lora_alpha=1. However, choosing 2*rank seems to destroy the model's performance. Perhaps bigger alpha is...

_not stale_

Thanks for reporting this @damian0815. I had forgotten about this PR. The issue should now be fixed, as well as a few other issues. If you find that LoRA for...

> definitely useful, yes. That's good to hear. Hopefully this PR can be merged some day so that we can have MHA support in PEFT proper, it's just that multihead...

Great, thanks for confirming @damian0815, and sorry for the wrong path. I tried to create a unit test based on the description you provided, I think I could reproduce your...

@elementary-particle Thanks for the update. Could you please run `make style`?

@elementary-particle This PR is almost good to go, just a small merge conflict, could you please check it out?

Thanks @iuliaturc for checking again. I merged the PR but will re-open this issue, as it's not fully resolved. Btw with the latest PEFT version, we added some [utilities](https://huggingface.co/docs/peft/main/en/developer_guides/troubleshooting#check-layer-and-model-status) that...