sd-webui-controlnet
sd-webui-controlnet copied to clipboard
Mikubill
Multiple control units in API
I got arbitrary multi-controlnet processing units to work through the api. You can now even use more than 100 processing units if you have the resources 😄 But I haven't implemented it in the img2img route yet, and some corner cases are not accounted for yet.
What I did here so far is, monkey patch the existing /sdapi/v1/txt2img to include an additional optional field controlnet_units that expects a list of control processing unit parameters. (which is mapped to the 14*n args packs the script expects as input)
I successfully reproduced the behaviour of the original route that does not run the process or postprocess of any alwayson script, except for controlnet.
Example request:
POST on route /sdapi/v1/txt2img:
(arrows <- to emphasis ellipsis)
{
"enable_hr": false,
"denoising_strength": 0,
"firstphase_width": 0,
"firstphase_height": 0,
"hr_scale": 2,
"hr_upscaler": "string",
"hr_second_pass_steps": 0,
"hr_resize_x": 0,
"hr_resize_y": 0,
"prompt": "",
"styles": [],
"seed": -1,
"subseed": -1,
"subseed_strength": 0,
"seed_resize_from_h": -1,
"seed_resize_from_w": -1,
"sampler_name": "Euler",
"batch_size": 1,
"n_iter": 1,
"steps": 50,
"cfg_scale": 7,
"width": 512,
"height": 512,
"restore_faces": false,
"tiling": false,
"negative_prompt": "",
"eta": 0,
"s_churn": 0,
"s_tmax": 0,
"s_tmin": 0,
"s_noise": 1,
"override_settings": {},
"override_settings_restore_afterwards": true,
"script_args": [],
"sampler_index": "Euler",
"controlnet_units": [
{
"input_image": "...", // <-
"module": "depth",
"model": "diff_control_sd15_depth_fp16 [978ef0a1]"
},
{
"input_image": "...", // <-
"module": "canny",
"model": "diff_control_sd15_canny_fp16 [ea6e3b9c]"
}
]
}
Response:
{
"images": [...], // <-
"parameters": {
"enable_hr": false,
"denoising_strength": 0,
"firstphase_width": 0,
"firstphase_height": 0,
"hr_scale": 2,
"hr_upscaler": "string",
"hr_second_pass_steps": 0,
"hr_resize_x": 0,
"hr_resize_y": 0,
"prompt": "",
"styles": [],
"seed": -1,
"subseed": -1,
"subseed_strength": 0,
"seed_resize_from_h": -1,
"seed_resize_from_w": -1,
"sampler_name": "Euler",
"batch_size": 1,
"n_iter": 1,
"steps": 50,
"cfg_scale": 7,
"width": 512,
"height": 512,
"restore_faces": false,
"tiling": false,
"negative_prompt": "",
"eta": 0,
"s_churn": 0,
"s_tmax": 0,
"s_tmin": 0,
"s_noise": 1,
"override_settings": {},
"override_settings_restore_afterwards": true,
"script_args": [],
"sampler_index": "Euler",
"script_name": null,
"controlnet_units": [
{
"input_image": "...", // <-
"mask": "",
"module": "depth",
"model": "diff_control_sd15_depth_fp16 [978ef0a1]",
"weight": 1,
"resize_mode": "Scale to Fit (Inner Fit)",
"lowvram": false,
"processor_res": 64,
"threshold_a": 64,
"threshold_b": 64,
"guidance": 1,
"guessmode": true
},
{
"input_image": "...", // <-
"mask": "",
"module": "canny",
"model": "diff_control_sd15_canny_fp16 [ea6e3b9c]"
"weight": 1,
"resize_mode": "Scale to Fit (Inner Fit)",
"lowvram": false,
"processor_res": 64,
"threshold_a": 64,
"threshold_b": 64,
"guidance": 1,
"guessmode": true
}
]
},
"info": "{\"prompt\": \"\", \"all_prompts\": [\"\"], \"negative_prompt\": \"\", \"all_negative_prompts\": [\"\"], \"seed\": 2134127272, \"all_seeds\": [2134127272], \"subseed\": 2639399611, \"all_subseeds\": [2639399611], \"subseed_strength\": 0.0, \"width\": 512, \"height\": 512, \"sampler_name\": \"Euler\", \"cfg_scale\": 7.0, \"steps\": 50, \"batch_size\": 1, \"restore_faces\": false, \"face_restoration_model\": null, \"sd_model_hash\": \"0fc198c490\", \"seed_resize_from_w\": -1, \"seed_resize_from_h\": -1, \"denoising_strength\": 0.0, \"extra_generation_params\": {\"ControlNet Enabled\": true, \"ControlNet Module\": \"depth\", \"ControlNet Model\": \"diff_control_sd15_depth_fp16 [978ef0a1]\", \"ControlNet Weight\": 1.0, \"ControlNet Guidance Strength\": 1.0}, \"index_of_first_image\": 0, \"infotexts\": [\"Steps: 50, Sampler: Euler, CFG scale: 7.0, Seed: 2134127272, Size: 512x512, Model hash: 0fc198c490, Seed resize from: -1x-1, Denoising strength: 0.0, ENSD: 31337, ControlNet Enabled: True, ControlNet Module: depth, ControlNet Model: diff_control_sd15_depth_fp16 [978ef0a1], ControlNet Weight: 1.0, ControlNet Guidance Strength: 1.0\"], \"styles\": [], \"job_timestamp\": \"20230225223609\", \"clip_skip\": 1, \"is_using_inpainting_conditioning\": false}"
}
A few questions I have:
- is monkey patching the existing webui routes a bad idea? is there a risk these routes get updated and the extension stops working because of that?
- should we add
scribble_modeandrgbbgrfields to control processing unit request params?
~~I am still looking for a way to reuse as much code as possible from the original sdapi/v1/*2img routes.~~ (see discussion below)
Updating the /controlnet/*2img routes by removing the spliced controlnet_* fields (except for controlnet_units) will break existing code. I think the best way to introduce this feature smoothly would be to deprecate the existing controlnet_* fields with a warning while still retaining both the controlnet_units and controlnet_* fields simultaneously.
Closes #314, closes #371
Update: I found a way to reuse the txt2img route code completely, while also monkey patching the original txt2img route to support an additional controlnet_units param.
Using our own /controlnet/*2img routes might be a better idea than updating the existing webui routes in place. Either way, there shouldn't be a problem if the webui updates the code of any existing route now.
To test:
- [x] Concurrent requests. I expect concurrent requests to be processed serially without any issue now.
- [x] Custom selectable script, like loopback or x/y/z grid. Possible that using a custom script offsets args passed to controlnet script.
/controlnet/img2img and controlnet/txt2img are both implemented now using the controlnet_units json property. The old controlnet_* json properties can still be used for backwards compatibility, however now there is a warning when using them in the request body, which redirects users to using controlnet_units:
The /docs route gives an example of how to use controlnet_units and now hides the old controlnet_* params:
The default value of controlnet_units is [], in which case both *2img controlnet routes have the exact same behavior as if calling the sdapi/v1/*2img routes.
The webui routes under sdapi/v1/ are not monkey patched in place anymore.
I'll mark as ready when I did a couple more tests on my side, i.e. I have not yet verified that custom scripts still work.
Alright I think I tested the main use-cases now. Hopefully that covers everything. I would be more confident that this doesn't break anything with automated testing.
By the way I could not find a better name for controlnet_units. Does it convey its intent?
If anyone has a better suggestion I'm all ears 👍 we probably won't be able to change it in the future without breaking the API.
Note that API params will change later for https://github.com/Mikubill/sd-webui-controlnet/pull/393
@ljleb @Mikubill
Hello, it seems that after updating to the latest main branch, the API /docs no longer shows the controlnet txt2img route, and parameters. It only shows two controlnet routes:
Do you have any idea on what may be wrong? I'm very eager to try the new multiple control units support via the API!
Thank you
Anyone is able to use 2 control nets with the new API yet? For me it looks like two control nets aren't working together, here is my payload
payload = {
"init_images": imgtext,
"sampler_name": "Euler a", "prompt": PROMPT, "alwayson_scripts": { "controlnet": { "args": [ { "input_image": imgraw, "module": "canny", "model": CANNY, "weight": 0.9, "threshold_a": 50, "threshold_b": 100, "guidance_start": 0, "guidance_end": 1 } , { # "input_image": imgraw, "module": "depth", "model": DEPTH # "weight": 0.9, # "threshold_a": 50, # "threshold_b": 100, # "guidance_start": 0, # "guidance_end": 1 } ] } } }
response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)
