Round renderings with dmtet finetuning

[mvp18]

Description

Hi @ashawkey , thanks for your great work!

I think this issue has been referenced before - here and here. And I have tried all solutions you have suggested - commenting out the torch.compile lines and running with different seeds, but the problem still persists with torch 2.x. Here are a few examples:

https://user-images.githubusercontent.com/24255665/233854048-c0119d73-576f-4d01-af57-5a3eb6166b58.mp4

generated with python main.py -O --text "a DSLR photo of an ice cream sundae" --workspace trial2_icecream --dmtet --iters 5000 --init_ckpt trial_icecream/checkpoints/df.pth

https://user-images.githubusercontent.com/24255665/233854110-f50e0ec3-324f-4815-ae83-40c5ad8deb04.mp4

generated with python main.py -O --text "a highly detailed stone bust of Theodoros Kolokotronis" --workspace trial2_stonehead --dmtet --iters 5000 --init_ckpt trial_stonehead/checkpoints/df.pth

What I noticed from the validation log that this rounding effect starts with the very initial images of the finetuning process. Like this:

I get this warning that the weights of the stable diffusion model are getting randomly initialized before the finetuning starts, but don't think that's the issue since your comments say the guidance model is not required during finetuning.

Steps to Reproduce

python main.py --workspace trial2_icecream -O --test python main.py --workspace trial2_stonehead -O --test

Expected Behavior

Can you share any inputs on how to fix this? Is it only a torch 2.x issue you think?

Environment

Debian GNU/Linux 11, Torch 2.0, CUDA 11.7

[ashawkey]

@mvp18 Hi, do you mean the first stage already produces a sphere? The first two videos don't seem like produced by DMTet.

[mvp18]

@ashawkey thank you so much for getting back. Yes, the first 2 videos are generated after dmtet finetuning. I think the results are similar to the issues 218 and 204.

And the round images you see above are from the validation logs in trial2_icecream/validation and trial2_stonehead/validation respectively. The ones before dmtet finetuning look perfectly fine as below:

https://user-images.githubusercontent.com/24255665/233933242-c18c4775-e9ba-4f0f-bd6b-3795024b2a77.mp4