Mahdi Beitollahi

icon location Canada icon location I am a machine learning scientist working on distributed machine learning and privacy-preserving machine learning methods such as Federated Learning.

Results 2 issues of Mahdi Beitollahi

supports learning the combination weights of pre-trained LoRA modules

6 comment

Based on #1655 Adds a use_wlora config to `LoraLayer` that allows learning the combination weights (i.e. `wlora_weights) of pre-trained LoRAs.

Learning the combination weights of pre-trained LoRA Modules

5 comment

### Feature request PEFT can combine pre-trained LoRA modules by averaging them or providing custom weights for weighted averaging. This [paper ](https://arxiv.org/abs/2402.15414) showed that learning these weights is better than...