stable-diffusion-webui-forge [Feature Request]: Implement X-Adapter

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

https://github.com/showlab/X-Adapter Code for X-Adapter is finally out.

X-Adapter enables plugins pretrained on old version (e.g. SD1.5) directly work with the upgraded Model (e.g., SDXL) without further retraining.

Is it possible to implement it into Forge?

Proposed workflow

Use 1.5 Loras on SDXL models.

Additional information

No response

Feb 17 '24 06:02 sashasubbbb

@lllyasviel What is the best way to run 2 unets side by side now? It seems like the core logic for X-Adapter is mapping sd15 hidden states to SDXL hidden states (Add to original SDXL hidden states) in decode part of unet.

https://github.com/showlab/X-Adapter/blob/d5460d3baaa3e995c18dbf6680a843c8a3a9b3f9/pipeline/pipeline_sd_xl_adapter_controlnet_img2img.py#L1088-L1227

Feb 18 '24 03:02 huchenlei

Awesome following this topic

Feb 19 '24 00:02 FurkanGozukara

@huchenlei Do they absolutely have to be side-by-side, or can they be loaded and unloaded one, then the other?

I ask because it doesn't want to run as-is on my 8GB of VRAM. It OOM'ed during the 2nd set of generation iterations. By the description / tutorial here by the author I was guessing it first generates using SD1.5 and then using SDXL, but I could be wrong.

I was able to make it work by changing every CUDA reference to CPU just for testing at a very slow 30~ minutes an image. Using inference.py --plugin_type "lora" with --adapter_guidance_start_list 0.7 and an "old pencil sketch" style LoRA, I get a decent effect.