abpani

@kartikayk Are you not able to clone the repo or download the model weight. for downloading the model weight use below command wget --header="Authorization: Bearer HF_TOKEN" https://huggingface.co/datasets/GeneralAwareness/Various/resolve/main/file.zip I am able...

Okay nevermind. Made it work after checking the source code . it did not work for the consolidated.pth but working for the HF checkpiint files . safetensors.

Finetuning is working now but only using safetensors. 'peak_memory_reserved': 12.637437952

@zzlgreat same error for me too.

and its normal peft finetuning not unsloth.

its just normal model loading Qwen2-7b-instruct-gptq-int8 with automodelforcasuallm torch 2.3.1 auto_gptq 0.7.1 flash-attn 2.5.8 peft 0.11.1 bitsandbytes 0.43.1 the model loading itself is not balanced behaves like above.

yes qlora

@qgallouedec No it does not I am using bnb 4 bit to load the model. as you can see above it only takes 9 gb and is distributed among 4...

accelerate 0.34.2 bitsandbytes 0.43.3 flash-attn 2.6.1 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.68 nvidia-nvtx-cu12 12.1.105...

I dont get any errors though. There is no error with batch size of 1. I get oom error when I do more than 1. Please let me know if...