bitsandbytes icon indicating copy to clipboard operation
bitsandbytes copied to clipboard

Published 20 hours ago • verified publisher icon TimDettmers

Merge LoRA into 405B

Open junzhang-zj opened this issue 1 year ago • 4 comments

I am blocked from merging LoRA into LLaMA-3.1-405B with INT8 of BNB, and the specific details are below. Is there any action that I can try? https://github.com/huggingface/peft/issues/2065#issue-2523808306

junzhang-zj avatar Sep 16 '24 05:09 junzhang-zj

Error positioning: peft/utils/integrations.py im, imt, SCim, SCimt, coo_tensorim = bnb.functional.double_quant(im)

junzhang-zj avatar Sep 16 '24 10:09 junzhang-zj

Hi @junzhang-zj Can you let us know what versions of bitsandbytes, transformers, and PEFT that you are using?

matthewdouglas avatar Sep 16 '24 14:09 matthewdouglas

@matthewdouglas I have tried these versions bitsandbytes (0.43.3), transformers (4.44.2/4.43.3), and peft (0.12.0).

junzhang-zj avatar Sep 17 '24 02:09 junzhang-zj

Any updates on this issue?

junzhang-zj avatar Sep 19 '24 01:09 junzhang-zj

This should be resolved in PEFT#2245 and bitsandbytes 0.45.0. Feel free to raise again if necessary.

matthewdouglas avatar Jan 22 '25 21:01 matthewdouglas