William Zeng

I have the same error trying to do QLora FSDP for `meta-llama/Llama-3.2-3B-Instruct`. I'm using the latest package versions: `pip install accelerate==1.0.0 transformers==4.45.2 trl==0.11.3 peft==0.13.1 bitsandbytes==0.44.1`. I tried the solution proposed...

Thanks for the analysis @BenjaminBossan! I'll just use the dtype coercion temporary fix for now while waiting for the root fix. If only the LoRA weights are float32, then your...

Also, in my codebase, reverting to `transformers==4.44.2` doesn't resolve the non-uniform dtype issue. I tested it with Llama 2 7B, Llama 3.1 8B, and Llama 3.2 3B.

Agree with 2. For 1, while I like your approach in the code you linked, I'm wondering if it doesn't work as well in this case. I imagine two cases:...

Appreciate everyone's detailed feedback! After my oncall, I'll switch to regex and address everyone's feedback before switching out of draft mode.

Hi Devam, thanks for the PR! I've converted it to a draft because it's missing a file. Once you add it back, could you please test-run evaluation and inference to...

Hi @devampatel03, could you confirm whether you're still working on this PR? If there aren't updates soon, I'll close this PR as clean-up, and you can continue to work on...

Unfortunately a real run is required to verify that this works. It should be possible to create a GCP account with $300 free credit: https://cloud.google.com/free?utm_source=google&utm_medium=cpc&utm_campaign=na-US-all-en-dr-bkws-all-all-trial-e-dr-1710134&utm_content=text-ad-none-any-DEV_c-CRE_665665924741-ADGP_Hybrid+%7C+BKWS+-+MIX+%7C+Txt-Google+Cloud-Google+Cloud+Free-KWID_43700081235769791-kwd-394768718298&utm_term=KW_google+cloud+free+credits-ST_google+cloud+free+credits&gad_source=1&gclid=Cj0KCQjw4cS-BhDGARIsABg4_J3q_HTL8-cX82u1D7BZnx0fEpy_y-yZ_tHedCp8iCBCuVF5uS2Eys8aAru9EALw_wcB&gclsrc=aw.ds&hl=en

Hi Devam, do you have any updates on the testing?

Closing this PR due to inactivity. Feel free to make another one if you're still working on this.