diffusers Add PAG support to SD1.5

We recently integrated PAG into diffusers! See this PR [here] (https://github.com/huggingface/diffusers/pull/7944) we added PAG to SDXL

we also want to add PAG support to SD1.5 pipelines! we will need:

[x] StableDiffusionPAGPipeline (assigned to @shauray8, PR https://github.com/huggingface/diffusers/pull/8725)
[ ] StableDiffusionPAGImg2ImgPipeline (assigned to @sajadn )
[ ] StableDiffusionPAGInpaintPipeline (assigned to @blaisedmello )
[ ] StableDiffusionControlNetPAGPipeline (assigned to @tuanh123789 )
[ ] StableDiffusionControlNetPAGImg2ImgPipeline (assigned to @Bhavay-2001 )

You should put it under the pag folder
you can use the implementation of SDXL PAG pipelines as a reference (see this PRhttps://github.com/huggingface/diffusers/pull/7944 and you can find all the sdxl pag pipelines here https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/pag)

you need to add AutoPipeline so that you can use this API to create it

   AutoPipelineForImage2Image.from_pretrained(repo_id, controlnet=controlnet, enable_pag=True ...)

tests and docs

If you are interested in working on this, Let me know which pipeline(s) you want to work on:)

Jun 26 '24 08:06 yiyixuxu

Hello, I would like to have a go at it. Can I work on the StableDiffusionPAGInpaintPipeline?

Jun 26 '24 08:06 blaisedmello

@blaisedmello sure!

Jun 26 '24 08:06 yiyixuxu

Looks rather interesting, adding support for StableDiffusionPAGPipeline.

Jun 26 '24 12:06 shauray8

Hey, I'll work on StableDiffusionPAGImg2ImgPipeline

Jun 26 '24 16:06 sajadn

Hi, can i take StableDiffusionControlNetPAGPipeline

Jun 27 '24 08:06 tuanh123789

@tuanh123789 sure!

Jun 27 '24 08:06 yiyixuxu

Hi @yiyixuxu, I think I can work on StableDiffusionControlNetPAGImg2ImgPipeline. Pls assign this to me.

Jun 27 '24 15:06 Bhavay-2001

@Bhavay-2001 sure!

Jun 27 '24 16:06 yiyixuxu

@yiyixuxu can i try ?

Jun 29 '24 22:06 AbhinavJangra29

Hi, I'd like to work on StableDiffusionPAGInpaintPipeline if it's still open

Aug 17 '24 05:08 darshil0805

@darshil0805 sure!

Aug 17 '24 07:08 yiyixuxu

Hi @yiyixuxu , @a-r-r-o-w. I have opened a PR for SD Inpainting PAG Pipeline, can you review and help me to test this pipeline?

Sep 07 '24 09:09 darshil0805

how do i know the layer names? only up mid down?

Sep 17 '24 11:09 crapthings

cc @a-r-r-o-w, maybe we should update doc? (did we?)

Sep 17 '24 19:09 yiyixuxu

how do i know the layer names? only up mid down?

The correct usage is to look at either the modeling code, or state dict keys, to see what the layer naming looks like. Then you can specify either the full layer names or partial layer names as valid regular expressions when initializing the pipeline (via pag_applied_layers). For more advanced usage, you can take a look at helper methods in the PAGMixin.

See the docs and examples usages here.

TLDR; Our naming convention follows the pattern of something_blocks.whatever or something_blocks.[layer_index} or blocks.{layer_index} or transformer_blocks.{layer_index}, so the easiest way to enable them without looking into the state dict or modeling code is to specify something like pag_applied_layers="blocks.(10|20|42)" or pag_applied_layers="mid"

cc @a-r-r-o-w, maybe we should update doc? (did we?)

I think we did, but I noticed that the introduction docs are out of sync with the current usage, and I'm not sure if the current example works. I'll test and open a PR soon

Sep 17 '24 21:09 a-r-r-o-w

HI @yiyixuxu I'd like to work on StableDiffusionPAGImg2ImgPipeline if it's still open. Thanks

Sep 19 '24 11:09 saikomals

@airookie17 I think I'm about to merge in this https://github.com/huggingface/diffusers/pull/9463

Sep 20 '24 22:09 yiyixuxu