invoke-ai
[enhancement]: workflow with multiple models/samplers
Is there an existing issue for this?
- [X] I have searched the existing issues
Contact Details
No response
What should this feature add?
I'm not sure if this is what is planned for the "nodes" section of the interface, but it would be nice to have a way to automate a workflow involving running an image generation through more than one model/sampler. I have found that there are some images that I can't generate with a single txt2img generation, but that I can get amazing results by generating an image using one set of parameters, then taking it to img2img and using a completely different set of parameters to push it toward a different style.
Example 1: some samplers have been shown to get to a usable image in much fewer steps than others. There might be time benefits to using one of these samplers for the early steps of image generation, then switching over to another that gives more desirable final results.
Example 2: I've also been thinking that there may be interesting benefits to be had by doing things like ramping the CFG scale up or down during sampling. This could be useful, for example, if you like the compositions you get by using a high CFG scale (determined early in the process), but visual texture produced by using a low CFG scale (determined later in the process), or vice versa.
Example 3: Similarly, the prompt could be changed mid-generation. This would make it possible to do things like prompt editing.
Example 4: This is where I've been thinking this might be really useful with the new Stable Diffusion v2 stuff, being able to switch models could allow you to get the benefits of the new models without loosing some of the things that people are so upset about them removing, like celebrity faces, artist styles, or nudity. It could be even more useful though when working with finetuned models.
If, for example, you want to get the style of one of nitrosocke's amazing dreambooth models mixed with a character or style found in some other finetuned model, currently, you have to either try to merge the models (which sometimes works and sometimes doesn't, and I keep a copy of automatic1111's interface around almost exclusively for this purpose) and hope that both finetunings survive the process. Instead, being able to run the early steps of generation with one model, then switch to another to finish the process would open up some interesting possibilities.
Alternatives
No response
Aditional Content
No response
Another, more specific example, the DPM samplers take about twice as long per step as the euler/ddim samplers, but seem to give more texture on things like skin. An obvious optimization is to do the early steps with euler, then switch to DPM for the last few steps. Of course, even better results might be achieved by doing an euler generation, upscaling, then doing a DPM pass over the image (probably in tiles, which I believe is how the hires_fix option works) using a DPM sampler.
This may also be related to the --karras_max option, but I'm having trouble finding more information on what that does than what's in the brief description on https://github.com/invoke-ai/InvokeAI/blob/main/docs/features/CLI.md