Benjamin Bossan

ping @githubnemo

With https://github.com/huggingface/transformers/pull/33361 being merged (which marks torchao as traininable), once the next transformers version is released (>4.44.2), the GPU tests on this PR should pass (I tested locally). This PR...

> is there any plan for support `NF4Tensor` AFAICT, torchao NF4 is not supported in transformers (which may change in the future). Therefore, I don't have plans to support it...

> I think maybe `transformers` already support it by passing the `AOBaseConfig` I looked this up but didn't see any config for NF4. Could you please paste a snippet that...

> Here is the `transformers` code for NF4 torchao: I see, thanks. It's not really straightforward to use, I hope this will be simplified in the future. > I implement...

ping @SunMarc Could I get a review please, it should hopefully not take long.

No worries, that happens sometimes. Thanks a lot for the review.

Did you use the `modules_to_save` feature to fully fine-tune those original weights? E.g.: `config = `LoraConfig(..., modules_to_save=["layer1", "layer2"])` makes it so that "layer1" and "layer2" are fully fine-tuned. PEFT creates...

> I'm directly tuning the original model for reducing memory consumption. I see, yes, using `modules_to_save` will incur a bit of extra memory. Maybe you could try manually moving the...

Just to be sure I understand correctly, you want to use DDP and you run out of memory? Could you please paste the full error message you get?