Benjamin Bossan

@ariG23498 Could you please run `make style` so that the CI can pass?

Thanks @ariG23498 for the latest fixes and @nbasyl for the review. I did a small test using [this DoRA script](https://github.com/huggingface/peft/blob/c039b00358c1c7d9139f92c510fb035ebfe753fa/examples/dora_finetuning/dora_finetuning.py) by calling: `CUDA_VISIBLE_DEVICES=0 time python dora_finetuning.py --quantize --lora_dropout 0 --use_dora`...

> Do you mean all the variants found [here](https://github.com/huggingface/peft/tree/main/src/peft/tuners/lora)? Also I think it would be better to have the current change made to DoRA only, and then create another PR...

Thanks for the update. Let's also add it here: https://github.com/huggingface/peft/blob/93ddb1015a637e72c6e61a82852c7bb127b13d66/src/peft/tuners/lora/tp_layer.py#L214-L221 The other layer types don't seem to properly implement DoRA yet, so we can keep those to a separate PR....

Thanks for the updates. I'll re-run the script later, as the first test was only very short, to get some final numbers to report.

Update: So I re-ran the script for a while longer and with a higher batch size (before it was 1), using: `$CUDA_VISIBLE_DEVICES=0 time python examples/dora_finetuning/dora_finetuning.py --quantize --lora_dropout 0 --batch_size 16...

FWIW, I can successfully fine-tune a prefix tuning model with Llama 2 and Llama 3.2 using the script from the [method comparison suite](https://github.com/huggingface/peft/tree/main/method_comparison/MetaMathQA). Therefore, I don't think there is a...

@Martin-Ndlovu Since you use the `adapters` package, your problem is unrelated to the PEFT package. They don't rely on PEFT but have their own implementations, so there is nothing we...

Thanks for opening the issue and suggesting to add this new method to PEFT. @githubnemo and I have already looked at sine-LoRA and we're currently discussing what the best way...

_not stale_