stable-diffusion-webui icon indicating copy to clipboard operation
stable-diffusion-webui copied to clipboard

[Feature Request]: In memory merging of models

Open 2x-y opened this issue 2 years ago • 0 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

There's an increase in models that offer an inpainting version. For the img2img tab, add an option to automatically merge any model with the standard inpainting model in memory to avoid the need for specific inpainting models. This could also be used for InstructPix2Pix models

For txt2img, an equivalent option could allow multiple models to be merged in memory for image generation. The merge models and ratios could be saved into the png data. This would be similar to the SuperMerger extension:

https://github.com/hako-mikan/sd-webui-supermerger

Proposed workflow

img2img tab

  1. Add an "Inpainting model" to the img2img tab. If selected, this will be merged with the selected model in memory without saving to file.
  2. Use the merged model for inpainting etc.
  3. Save the merged model data to the png data

txt2img tab

  1. Allow multiple models to be optionally selected together with merge rations and merge mode.
  2. Merge the models in memory before generating the image
  3. Save the merged model data to the png data

Alternatively, produce a common merge model section that can be shared by both for both txt2img and img2img (the end result would be the same but offering a specific inpainting model on img2img may be more user-friendly).

Additional information

No response

2x-y avatar Apr 19 '23 08:04 2x-y