threestudio
threestudio copied to clipboard
Is it possible to make SDXL guidance work?
I still hope to check if anyone makes SDXL guidance work. I implemented one but it did not work. The major changes I made was the prompt process part as SDXL needs text embedding from two text encoders as well as pooled embedding and the Unet to encode image size information.
There are 2 weird things I noticed:
- If I set the data type of the VAE encoder to be float16, soon the gradient becomes nan, which is different from sd 2.0.
- If I set the data type of VAE encoder to be float32, I no longer have the nan issue but I can only get very blurry results at the beginning and it diverges as the training goes on.
Looking forward to any thoughts and suggestions! Thanks in advance!
You can take some inspiration from the repo pinned in my profile.