Juan A. Rodriguez
Juan A. Rodriguez
@kaihe Did you figure out how to fix the NaN issue? Having the same NaN loss problem.
It is set to 1000 by default by pytorch lightning. You need to set it in the .yaml file, with something like: ``` lightning: trainer: (...) max_epochs: 2000 ```
I'm facing exactly the same problem, I see that you fix the error by disabling gradient checkpointing (https://github.com/rinongal/textual_inversion/blob/0a950b482d2e8f215122805d4c5901bdb4a6947f/ldm/modules/diffusionmodules/util.py#L112). Cool! Hope we find a way to make it compatible though.
Same issue here, any updates?
Using CPU instead of GPU would not work. Use smaller image sizes.
@chinoll I have a similar problem. Where do you exaclty add the .../bin/ninja path in the torch/utils/cpp_extension.py file?
That worked, thanks a lot @chinoll
Hi @Winnie202, awesome results!! I'm glad you could reconstruct the small texts. We could try generating synthetic scene-text as a follow-up
Next PR will fix this issues. Thanks all for the effort 💕
It seems to me like an error with flash-attention. can you try installing flash-attn separately? Checking this on my side, will update the installation steps