threestudio
threestudio copied to clipboard
ImageDreamfusion accumulate grad, guidance_eval in sd,if
-
Accumulates grad in imagedreamfusion Fixes imagedreamfusion to accumulate grads from ref and guidance
-
Adds guidance_eval to stable_diffusion, deep_floyd Adds the ability to optionally evaluate guidance periodically, in stable_diffusion and deep_floyd (as done in zero123)
This does not affect current working of stable_diffusion, deep_floyd, it only adds the option to visualize the guidance.
Example
I used the following command to check the results with stable diffusion guidance:
python launch.py --config configs/experimental/imagecondition.yaml --train --gpu 0 system.freq.ref_only_steps=20 system.freq.guidance_eval=13
To run with deep floyd guidance, I uncommented the requisite lines in configs/experimental/imagecondition.yaml
.
First, since ref_only_steps=20
, it should run with just rgb_loss for 20 iterations.
Then, from 21st iteration, it should also compute guidance loss.
Then, since freq.guidance_eval=13
, at the 26th iteration, it should save the results of guidance_eval:
With stable-diffusion:
With deep-floyd:
@bennyguo this introduces redundancies but I think it's a good short-term step... so unless you have a concern, we'd like to merge it today. Later I recommend we fold some of the loss calculation/logging code in the base system class and maybe fold some of the eval guidance logic in the base guidance class, to avoid unnecessary code duplication. How does that sound?
I've folded guidance_eval_save into base.py, and address the comments. Please check!
Tested! Merging with main