ReNoise-Inversion icon indicating copy to clipboard operation
ReNoise-Inversion copied to clipboard

Inversion with guidance scale > 1.0

Open M4xim4l opened this issue 10 months ago • 5 comments

Hi,

I noticed that in the default run config, the guidance scale for reconstruction is set to 0. and the examples use a guidance scale of 1.0 for inference. I tried to set that to the usual value of 7 and get the error:

Traceback (most recent call last):
  File "/mnt/USER/ReNoise-Inversion-main/inversion_example_sd.py", line 30, in <module>
    _, inv_latent, _, all_latents = invert(input_image,
  File "/mnt/USER/ReNoise-Inversion-main/main.py", line 44, in run
    res = pipe_inversion(prompt = prompt,
  File "/mnt/USER/ReNoise-Inversion-main/src/pipes/sd_inversion_pipeline.py", line 153, in __call__
    latents = inversion_step(self,
  File "/mnt/USER/ReNoise-Inversion-main/src/renoise_inversion.py", line 146, in inversion_step
    noise_pred = noise_regularization(noise_pred, noise_pred_optimal, lambda_kl=pipe.cfg.noise_regularization_lambda_kl, lambda_ac=pipe.cfg.noise_regularization_lambda_ac, num_reg_steps=pipe.cfg.noise_regularization_num_reg_steps, num_ac_rolls=pipe.cfg.noise_regularization_num_ac_rolls, generator=generator)
  File "/mnt/USER/ReNoise-Inversion-main/src/renoise_inversion.py", line 11, in noise_regularization
    l_kld = patchify_latents_kl_divergence(_var, noise_pred_optimal)
  File "/mnt/USER/ReNoise-Inversion-main/src/renoise_inversion.py", line 69, in patchify_latents_kl_divergence
    kl = latents_kl_divergence(x0, x1).sum()
  File "/mnt/USER/ReNoise-Inversion-main/src/renoise_inversion.py", line 82, in latents_kl_divergence
    torch.log((var1 + EPSILON) / (var0 + EPSILON))
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 0

Is it possible to use ReNoise inversion with classifier-free guidance (guidance scale > 1) and why do you use a value of 0 for guidance during the inversion and 1.0 during inference in the examples?

Thank you!

M4xim4l avatar Apr 24 '24 17:04 M4xim4l