Benjamin Bossan
Benjamin Bossan
Thanks for investigating further. `modules_to_save` is intended to work with DeepSpeed, so if it doesn't, we should try to fix it. Again, if you could provide more details as mentioned...
> Yes, that’s right. Stage 1 trained successfully (the main issue there was with resize_token_embeddings, which I fixed manually). > However, I still face GPU OOM issues whenever modules_to_save is...
@mbaddar1 Just asking if you still plan on working on this.
Hi, no worries, there is no time pressure. I just wanted to ping you, in case you simply forgot about this. Take all the time you need.
Thanks for this PR that proposes to add Weight LoRA to PEFT. Do you have a link to the full paper? I only skimmed the implementation, but from what I...
> These constraints should not be taken into account in the WeightLoRA method, but in the implementation of the optimizer step (e.g. SGD with projection). In our paper, we provide...
_not stale_
Thanks for coming up with this proposal. Indeed, this is something we have on our backlog for a long time. As you can imagine, providing objective and useful information on...
> I would be interested to contribute as well Thanks for the offer. As mentioned, as a first step, we could use some help with updating the "blurbs" of the...
> How about having a sample fine-tuning script for each method and comparing different approaches for different tasks? I'm not 100% sure what you mean, but let's start with a...