Using both LORA and FSDP results in error

[shridharsamantaroy]

🐛 Bug

Setting both LORA and FSDP options to true while fine tuning results in

ValueError: FlatParameter requires uniform dtype but got torch.float16 and torch.float32

To Reproduce

Run an experiment with the OASST data set (https://www.kaggle.com/code/philippsinger/openassistant-conversations-dataset-oasst1?scriptVersionId=126228752) with both LORA and FSDP turned on

I have also attached the experiment configuration and logs logs_osst-example-fsdp.zip

[moseshu]

I have same errors .Did you solve it?

[maxjeblick]

Thank you for bringing up this issue!

The occurrence of the ValueError can be attributed to selecting float16 as the Backbone Dtype. This could be fixed by changing the data type to float32. Yet, another issue arises: ValueError: FlatParameter requires uniform requires_grad, which seems to be a bug in the torch library (see here).

Therefore, it appears that FSDP is currently incompatible with LORA. I'll monitor this issue and push a fix once available (or this will be fixed in peft/torch libraries, ideally).