custom pipelines and from_pretrained

[WASasquatch]

What API design would you like to have changed or added to the library? Why?

custom pipelines is available through the master from_pretrained() method, but do not work for other pipelines. It would be beneficial if you could load custom features into any pipeline, such as the much needed and missing functionality of img2img and inpainting for custom pipelines.

What use case would this enable or better enable? Can you give us a code example?

It wouldn't make custom pipelines a novelty, but a functioning part of diffusers that could be used in production services, and production workflows.

Currently if you tried to load custom pipes with say StableDiffusionImg2ImgPipeline() you will get errors about cache_download and also Could not locate the pipeline.py inside clip_guided_diffusion. (or whatever community pipe)

It would be cool, still, if custom pipes were just plugins. So you could do something like custom_pipelines = ['clip_guided_diffusion', 'weighted_stable_diffusion', 'etc'] sort of deal to add multiple features.

[WASasquatch]

I have been trying to get clip guidance into img2img and inpaint for weeks now, to no avail, and it's getting super frustrating. I personally find text2img as a novelty and only a stepping stone to production workflows, and I do not like how community pipelines focuses on DiffusionPipeline and just text2img.

[patrickvonplaten]

Thanks for the issue. I think it'll be quite difficult, technically, trying to ensure that one can load multiple custom pipelines into one given that they all define their own step function and also rely on different weights, e.g. clip guided diffusion relies on more weight parameters than stable diffusion text2image. The best I can think of at the moment is doing something like:

from diffusers import DiffusionPipeline

stable_diffusion = DiffusionPipeline.from_pretrained(model_id)
clip_guided = DiffusionPipeline.from_pretrained(model_id, custom="clip_guided_diffusion", **stable_diffusion.components)  # this won't allocate new memory

In the long run we should maybe really try to have a better compatibility with AUTOMATIC1111 which is really nice (see PR/issue here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2577)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[WASasquatch]

And don't you think that may have been a short sight in "community" development and usage through the API? It seems more intuitive for custom pipelines to be modules to a stable Diffusion pipeline. Which ever you have active by name, eg lpw_stable_diffusion would have a "step_extension" that is hooked into a main step function, like a override. That way you don't have to share components or configs, but just declare the pipe type.

Further more, since these are community and often have unique setup, it should be common practice for pipe builders to contain their setup methods. For example with clip guided there should be a simple pipe.set_clip_model( ID ) method rather than from pretrained setups etc. This way you aren't bloating conditions for certain pipes beyond simply "is it clip guided? Then setting model id is valid".

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.