Mugheeera comments

Results 5 comments of


                                            Mugheeera

CUDA OOM during ckpt saving for Llama2-70b

@HamidShojanazeri Thank you very much for your reply. I tried with a new install of `transformers 4.39.0.dev0` from the src but still encountered the same issue. I have also attempted...

CUDA OOM during ckpt saving for Llama2-70b

I tried with the following package versions and a fresh install of llama-recipes, but still getting the same error. `accelerate 0.28.0, appdirs 1.4.4, bitsandbytes 0.43.0, black 24.3.0, datasets 2.18.0, fire...

peft_method works fine with lora, but pops error when using prefix and llama_adapter

Hey, Did you manage to solve this? I am getting the same error.

peft_method works fine with lora, but pops error when using prefix and llama_adapter

`use_orig_params=False` results in this error. `[rank0]: RuntimeError: The size of tensor a (4096) must match the size of tensor b (4126) at non-singleton dimension 3` when `--peft_method prefix` and `ValueError:...

OOM when saving model, seems the clause "FSDP(model,...)" will cause the OOM.

Hey @cyrishe , Did you manage to solve this issue? I am encountering the same problem while trying to finetune 70B with FSDP and PEFT on 5 A100 x 80GB....