diffusers icon indicating copy to clipboard operation
diffusers copied to clipboard

Add Diffusers supported callbacks

Open DN6 opened this issue 10 months ago • 4 comments

Is your feature request related to a problem? Please describe. We currently support passing in callbacks to pipelines in order to manipulate/customise aspects of the diffusion process. It would be nice to have a set of officially supported callbacks in Diffusers that would allow us to support functionality such as turning off classifier free guidance out of the box.

Describe the solution you'd like. A potential design to support callbacks (this is just a pseudo-code example)

from diffusers import StableDiffusionPipeline
from diffusers.pipelines.callbacks import GuidanceCallback

pipe = StableDiffusionPipeline.from_pretrained("...")

callbacks = [GuidanceCallback()]
pipe.add_callbacks(callbacks)

image = pipe(...).images[0]

Additional context. Transformers does support callbacks: https://huggingface.co/docs/transformers/main/en/main_classes/callback for training, while ours I think would be mostly focused on inference.

DN6 avatar Apr 22 '24 09:04 DN6

let's start with something simple and just start to support some really useful callback functions with the current callback design

cc @asomoza here - can you take a lead on this?

yiyixuxu avatar Apr 22 '24 16:04 yiyixuxu

yes, I'll take this

asomoza avatar Apr 22 '24 17:04 asomoza

Hi @asomoza, @yiyixuxu Is this issue open for community?

Bhavay-2001 avatar Apr 24 '24 14:04 Bhavay-2001

@Bhavay-2001 Not for now, I'm doing it with a specific simple design so we don't change that much of what we have right now.

We're open to suggestions if you consider something important or want to contribute, here's the PR: https://github.com/huggingface/diffusers/pull/7761

asomoza avatar Apr 29 '24 17:04 asomoza