DHS-LLM-Workshop icon indicating copy to clipboard operation
DHS-LLM-Workshop copied to clipboard

Published 20 hours ago • verified publisher icon pacman100

Problem training with FSDP

Open agokrani opened this issue 1 year ago • 2 comments

When I am trying to train a model with FSDP, I am getting following error.

*** TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

It is happening on this specific line trainer.model = trainer.model_wrapped = FSDP(trainer.model, **kwargs)

and after a bit of debugging it feels like it has something to do with auto_wrap_policy. I am not really sure how to solve this. Do you have any suggestions. It was working fine until few days ago.

agokrani avatar Jan 03 '24 13:01 agokrani