Mika Sie

I have the same issue! Are you any further yet with solving this? My setup is a bit different but I think the settings that differ shouldn't have an effect...

Hi! I had the same issue but also some other ones. I did find what the issue was for this value error specifically. It was a while ago for me...

@muellerzr I have found an issue with ```accelerate test```, it seems like the issue with my training script has something to do with transformers: I saw that ```fsdp_transformer_layer_cls_to_wrap``` is [depreciated](https://discuss.huggingface.co/t/how-to-create-the-fsdp-config-json-file-for-trainer/41685/2)....

Thanks for your response! I have just tested the [Huggingface Bitsandbytes Guide](https://huggingface.co/docs/bitsandbytes/main/en/fsdp_qlora) . This script seems to work (and wow it is much better than mine haha). I use LLama3-8B...

Hi! I've added the print statements in the line you asked. The print staments are as follows: ``` def set_auto_wrap_policy(self, model): from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy, transformer_auto_wrap_policy default_transformer_cls_names_to_wrap = ( ",".join(model._no_split_modules)...

> I'm confused, first you reference the bnb guide but then you mention the PEFT guide. Which one was the one that worked? This might be of interest for future...

Here I am again haha! But I have good news, I found the issue with my training script: **The wandb callback caused the ValueError because it sets the model to...