cads support

[loboere]

https://arxiv.org/abs/2310.17347

[DN6]

@loboere could you share some of the results/benefits from this method?

[apolinario]

CADS is a technique that greatly increases diversity of generated images by adding scheduled noise to the conditioning at inference time.

Here's an implementation of it as an AUTOMATIC1111 extension: https://github.com/v0xie/sd-webui-cads

[yiyixuxu]

is it a new scheduler?

[Msadat97]

is it a new scheduler?

I'm glad to see that you are interested in our paper! CADS is not technically a new scheduler but a technique that can be used on top of common diffusion schedulers (like DDPM or DDIM) to increase the diversity of the outputs.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

@Msadat97 I just finished reading the paper a few hours ago and I understand how to implement it. The A1111 results look great! Thanks for your awesome work.

To summarize the process:

• Noise is added to conditioning vector every step of the inference process
• As inference progresses towards completion, lesser and lesser noise is added
• The resulting outputs are much more diverse in comparison to static guidance scale and dynamic guidance scale as experimented on DDPM

I believe this is a great way to improve diversity, but noticed while experimenting that prompt-following or adherence to subject image is somewhat lost (due to the noisy signal addition in conditioning). You highlight this for class-conditional imagenet generation as a minor loss in accuracy. For general SD though, it seems to be having a slightly higher impact.

@yiyixuxu May I work on this? I have a feeling that the functionality could be implemented with callback_on_step_end combined with a few minor changes adding support for a before-inference call to callback_on_step_end as done here. Not sure though, will need to try sometime this weekend.

[Msadat97]

The diversity increase might be more obvious for models that are trained on smaller datasets compared to SD. What we mainly observed was that for example if the prompt is kind of leading to a repetitive result (like the astronaut on the moon), then CADS can alleviate this issue quite a bit. Also for prompt adherence, I think one has somehow control over that by shifting the cads noise schedule more toward the beginning of the sampling (i.e. choosing tau_1 closer to 1). In the extreme case of tau_1=1, we get the default sampling back. We also noticed that deterministic samplers generally get more diverse with less noise.

I'd be happy to answer any other question you might have regarding the paper. Thanks for taking the initiative.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.