sd-webui-controlnet
sd-webui-controlnet copied to clipboard
Mikubill
[Bug]: Controlnet API not being used
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
What happened?
I have configured my JSON payload in the format shown in the example of the API docs, but it simply isn't using controlnet. It is passing the "prompt" to the txt2img endpoint and generating off that only. I have updated controlnet and webui. I can see in the console that the canny model is not being loaded. Here is how my json payload is being passed:
encoded_image = base64.b64encode(image_bytes).decode('utf-8')
request_data = {
"prompt": prompt,
"negative_prompt": "",
"seed": -1,
"subseed": -1,
"subseed_strength": 0,
"batch_size": 1,
"n_iter": 1,
"steps": 15,
"cfg_scale": 7,
"eta": 0,
"alwayson_scripts": {
"controlnet": {
"args": [
{
"input_image": [encoded_image],
"module": "canny",
"model": "control_v11p_sd15_canny [d14c016b]",
#"controlnet_guidance": 1.0
}
]
}
}
}
Steps to reproduce the problem
async with aiohttp.ClientSession() as session:
async with session.post("http://localhost:7860/sdapi/v1/txt2img", json=request_data, headers=headers) as response:
response_data = await response.json()
What should have happened?
I expect to see it load the canny model in the console and use it. It is not.
Commit where the problem happens
webui: 0cc0ee1b controlnet: c9c8ca6e (Sun May 7 17:15:00 2023)
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
--autolaunch --xformers --listen --enable-insecure-extension-access --api
Console logs
No errors in the console, it's just not using controlnet
Additional information
No response
Same exact issue. I'm not sure what could be causing it, as using this same format was working previously.
I am not familiar about API is this because users enable multi controlnets but only give one unit in API? can anyone test it and see do we update codes or docs?
I did have multiple controlnets enabled in webui, but I set it back to 1 and still the same issue.
"input_image" is string not List, like "input_image": "data:image/jpeg;base64,xxxx"
"input_image" is string not List, like
"input_image": "data:image/jpeg;base64,xxxx"
I tried passing it as a string instead of a list, but no change: "input_image": encoded_image,
Hello we added example to API wiki
Below is a minimal working example for sanity check (this example is tested every day and you can trust that it should always work.)
import io
import cv2
import base64
import requests
from PIL import Image
# A1111 URL
url = "http://127.0.0.1:7860"
# Read Image in RGB order
img = cv2.imread('your_image.jpg')[:, :, ::-1]
# Encode into PNG and send to ControlNet
retval, bytes = cv2.imencode('.png', img)
encoded_image = base64.b64encode(bytes).decode('utf-8')
# A1111 payload
payload = {
"prompt": 'a handsome man',
"negative_prompt": "",
"batch_size": 1,
"steps": 20,
"cfg_scale": 7,
"alwayson_scripts": {
"controlnet": {
"args": [
{
"input_image": encoded_image,
"module": "canny",
"model": "control_v11p_sd15_canny [d14c016b]",
}
]
}
}
}
# Trigger Generation
response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)
# Read results
r = response.json()
result = r['images'][0]
image = Image.open(io.BytesIO(base64.b64decode(result.split(",", 1)[0])))
image.save('output.png')
If you cannot get it works with this standard sanity check, please attach your full log and a1111 config.json and your full launching cmd flags
I ran exactly this test script (except for changing input file name to a valid one), but it still did not engage canny model or use CN. It just generated an image based off prompt only. Can you enlighten me where the logging for SD API is kept? I enabled the --api-log flag, but could not find where the log was saved. My full launching command flags are "--autolaunch --xformers --listen --enable-insecure-extension-access --api --api-log". Here is my a1111 config.json:
{
"samples_save": true,
"samples_format": "png",
"samples_filename_pattern": "",
"save_images_add_number": true,
"grid_save": true,
"grid_format": "png",
"grid_extended_filename": false,
"grid_only_if_multiple": true,
"grid_prevent_empty_spots": false,
"n_rows": -1,
"enable_pnginfo": true,
"save_txt": false,
"save_images_before_face_restoration": false,
"save_images_before_highres_fix": false,
"save_images_before_color_correction": false,
"jpeg_quality": 80,
"export_for_4chan": true,
"img_downscale_threshold": 4.0,
"target_side_length": 4000,
"use_original_name_batch": true,
"use_upscaler_name_as_suffix": false,
"save_selected_only": true,
"do_not_add_watermark": false,
"temp_dir": "",
"clean_temp_dir_at_start": false,
"outdir_samples": "",
"outdir_txt2img_samples": "outputs/txt2img-images",
"outdir_img2img_samples": "outputs/img2img-images",
"outdir_extras_samples": "outputs/extras-images",
"outdir_grids": "",
"outdir_txt2img_grids": "outputs/txt2img-grids",
"outdir_img2img_grids": "outputs/img2img-grids",
"outdir_save": "log/images",
"save_to_dirs": true,
"grid_save_to_dirs": true,
"use_save_to_dirs_for_ui": false,
"directories_filename_pattern": "[date]",
"directories_max_prompt_words": 8,
"ESRGAN_tile": 192,
"ESRGAN_tile_overlap": 8,
"realesrgan_enabled_models": [
"R-ESRGAN 4x+",
"R-ESRGAN 4x+ Anime6B"
],
"upscaler_for_img2img": null,
"face_restoration_model": "CodeFormer",
"code_former_weight": 0.5,
"face_restoration_unload": false,
"show_warnings": false,
"memmon_poll_rate": 8,
"samples_log_stdout": false,
"multiple_tqdm": true,
"print_hypernet_extra": false,
"unload_models_when_training": false,
"pin_memory": false,
"save_optimizer_state": false,
"save_training_settings_to_txt": true,
"dataset_filename_word_regex": "",
"dataset_filename_join_string": " ",
"training_image_repeats_per_epoch": 1,
"training_write_csv_every": 500,
"training_xattention_optimizations": false,
"training_enable_tensorboard": false,
"training_tensorboard_save_images": false,
"training_tensorboard_flush_every": 120,
"sd_model_checkpoint": "dreamshaper_5BakedVae.safetensors [6d492d946c]",
"sd_checkpoint_cache": 0,
"sd_vae_checkpoint_cache": 0,
"sd_vae": "Automatic",
"sd_vae_as_default": true,
"inpainting_mask_weight": 0,
"initial_noise_multiplier": 1.0,
"img2img_color_correction": false,
"img2img_fix_steps": false,
"img2img_background_color": "#ffffff",
"enable_quantization": false,
"enable_emphasis": true,
"enable_batch_seeds": true,
"comma_padding_backtrack": 20,
"CLIP_stop_at_last_layers": 1,
"upcast_attn": false,
"use_old_emphasis_implementation": false,
"use_old_karras_scheduler_sigmas": false,
"no_dpmpp_sde_batch_determinism": false,
"use_old_hires_fix_width_height": false,
"interrogate_keep_models_in_memory": false,
"interrogate_return_ranks": false,
"interrogate_clip_num_beams": 1,
"interrogate_clip_min_length": 24,
"interrogate_clip_max_length": 48,
"interrogate_clip_dict_limit": 1500,
"interrogate_clip_skip_categories": [],
"interrogate_deepbooru_score_threshold": 0.5,
"deepbooru_sort_alpha": true,
"deepbooru_use_spaces": false,
"deepbooru_escape": true,
"deepbooru_filter_tags": "",
"extra_networks_default_view": "cards",
"extra_networks_default_multiplier": 1.0,
"sd_hypernetwork": "None",
"return_grid": true,
"do_not_show_images": false,
"add_model_hash_to_info": true,
"add_model_name_to_info": true,
"disable_weights_auto_swap": true,
"send_seed": true,
"send_size": true,
"font": "",
"js_modal_lightbox": true,
"js_modal_lightbox_initially_zoomed": true,
"show_progress_in_title": true,
"samplers_in_dropdown": true,
"dimensions_and_batch_together": true,
"keyedit_precision_attention": 0.1,
"keyedit_precision_extra": 0.05,
"quicksettings": "sd_model_checkpoint, inpainting_mask_weight",
"ui_reorder": "inpaint, sampler, checkboxes, hires_fix, dimensions, cfg, seed, batch, override_settings, scripts",
"ui_extra_networks_tab_reorder": "",
"localization": "None",
"show_progressbar": true,
"live_previews_enable": true,
"show_progress_grid": true,
"show_progress_every_n_steps": 10,
"show_progress_type": "Approx NN",
"live_preview_content": "Prompt",
"live_preview_refresh_period": 1000,
"hide_samplers": [],
"eta_ddim": 0.0,
"eta_ancestral": 1.0,
"ddim_discretize": "uniform",
"s_churn": 0.0,
"s_tmin": 0.0,
"s_noise": 1.0,
"eta_noise_seed_delta": 0,
"always_discard_next_to_last_sigma": false,
"postprocessing_enable_in_main_ui": [],
"postprocessing_operation_order": [],
"upscaling_max_images_in_cache": 5,
"disabled_extensions": [],
"sd_checkpoint_hash": "6d492d946ca8e128bc61da13b7e931778c0ca45574000a59dc92127f8debdf01",
"ldsr_steps": 100,
"ldsr_cached": false,
"SWIN_tile": 192,
"SWIN_tile_overlap": 8,
"sd_lora": "None",
"lora_apply_to_outputs": false,
"control_net_model_config": "models\\cldm_v15.yaml",
"control_net_model_adapter_config": "models\\sketch_adapter_v14.yaml",
"control_net_detectedmap_dir": "detected_maps",
"control_net_models_path": "",
"control_net_max_models_num": 1,
"control_net_model_cache_size": 5,
"control_net_control_transfer": false,
"control_net_no_detectmap": false,
"control_net_detectmap_autosaving": false,
"control_net_only_midctrl_hires": true,
"control_net_allow_script_control": true,
"control_net_skip_img2img_processing": false,
"control_net_monocular_depth_optim": false,
"control_net_only_mid_control": false,
"control_net_cfg_based_guidance": false,
"control_net_sync_field_args": false,
"control_net_modules_path": "",
"controlnet_show_batch_images_in_ui": false,
"controlnet_increment_seed_during_batch": false
}
Again, appreciate the help!
I tested your launch cmd and config file. It works without problem. The only possiblility is that your model hash mismatch make sure it is d14c016b if you use the test code
I tested your launch cmd and config file. It works without problem. The only possiblility is that your model hash mismatch make sure it is d14c016b if you use the test code
Yes, my canny model hash does match what's in the test code.
Are you using CN 1.1.150 with webui commit 0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8? Curious if there anything different about my setup.
I'll keep digging. Let me know if you have any other suggestions. Thanks
The issue seem pretty deep.
On line 934 in controlnet.py we have:
units = external_code.get_all_units_in_processing(p)
which is the call to the method that gives you the controlnet units. These are required to get the enabled controlnet units.
In automatic1111 (line 352) or vladmantic (line 268) modules/api/api.py, we have this:
args.pop('script_name', None)
args.pop('script_args', None) # will refeed them to the pipeline directly after initializing them
which is something that would give a heart attack to anyone who likes Haskell...
Now, modules/scripts.py : For automatic1111, we have on line 406:
script_args = args[script.args_from:script.args_to]
processed = script.run(p, *script_args)
For vlad, we have on line 364:
parsed = p.per_script_args.get(script.title(), args[script.args_from:script.args_to])
log.debug(f'Script run: {script.title()}')
processed = script.run(p, *parsed)
So, the thing is that when you call script.run(p, *parsed), you don't have controlnet scripts available anymore, because they got removed in api.py.
Finally, what happens is that when you finally get to controlnet.py process method, on line 985:
self.enabled_units = self.get_enabled_units(p)
p.script_args have been replaced by whatever values the selected script you were using had, for example values from outpainting script, and...
The actual values that can be seen by controlnet at this point, if you call it from the API, are not longer available in p.script_args, but are available in the local variable args.
So when later, in get_enabled_units, we get a call to get_all_units_in_processing(p), args is not passed and since p.script_args have been compromised at this point, it doesn't work as intended.
I'm not completely sure how this is related to the issue of @F0xbite, as he doesn't seem to be using a selectable optional script in his payload, though. But this issue I'm describing is definitively present for cases when you also have a selectable optional script active.
I still cannot understand but I will fix as soon as someone provide any instruction that I can reproduce the problem on my end. Rightnow it works very well on my PC. Tested everyday on multiple machines.
But I remember that some other extensions may cause some conflicts. perhaps you can try to disable all other extensions and see if that is the case
I just tried with all my extensions disabled (not the built-in ones), using vladmantic ui, and it still doesn't work.
Try this. It should help you reproduce the problem.
import json
import cv2
import numpy as np
import requests
import io
import base64
from PIL import Image, PngImagePlugin
url = "http://127.0.0.1:7860"
img = cv2.cvtColor(cv2.imread('somefile.png')[:, :, ::-1], cv2.COLOR_RGB2BGR)
retval, bytes = cv2.imencode('.png', img)
encoded_image = base64.b64encode(bytes).decode('utf-8')
payload = {
"prompt": "whatever",
"steps": 20,
"denoising_strength": 1,
"inpaint_full_res": True,
"inpaint_full_res_padding": 32,
"init_images": [encoded_image],
"styles": ["whatever style if you have it"],
"seed": 3787024609,
"sampler_name": "Euler a",
"cfg_scale": 7,
"alwayson_scripts": {
"controlnet": {
"args": [
{
"module": "inpaint_global_harmonious",
"model": "control_v11p_sd15_inpaint [ebff9138]",
"control_mode": "ControlNet is more important",
}
]
}
},
"script_name": "Outpainting",
"script_args": [
None, # unused
256, # pixels
8, # mask_blur
["up"], # direction
1, # noise_q
0.05, # color_variation
]
}
response = requests.post(url=f'{url}/sdapi/v1/img2img', json=payload)
r = response.json()
for i in r['images']:
image = Image.open(io.BytesIO(base64.b64decode(i.split(",",1)[0])))
png_payload = {
"image": "data:image/png;base64," + i
}
response2 = requests.post(url=f'{url}/sdapi/v1/png-info', json=png_payload)
pnginfo = PngImagePlugin.PngInfo()
pnginfo.add_text("parameters", response2.json().get("info"))
image.save('output.png', pnginfo=pnginfo)
@lllyasviel can you give a minimal working API code for multicontrolnet? I dont understand how to pass on more than one model inputs to the api.
I rolled back the old commit, in the 9eeb71 commit the example code is working, but in the latest commit the example code is not working, maybe some changes during this period made it not work, hope it can help ,Thanks.
@CikeyQi Does multicontrolnet also work? If so can you share an example?
I rolled back the old commit, in the 9eeb71 commit the example code is working, but in the latest commit the example code is not working, maybe some changes during this period made it not work, hope it can help ,Thanks. @CikeyQi Very strange, I just rolled back to commit 9eeb71 and it still isn't using controlnet, even using the test script.
Multicontrolnet, you would do something like this :
"alwayson_scripts": {
"controlnet": {
"args": [
{
"module": ...,
"model": ...,
},
{
"module": ...,
"model": ...,
},
{
"module": ...,
"model": ...,
}
]
}
},
Hey Guys, I have added a working multi controlnet inpainting API example in this issue. This is working as of 15th May 2023 with the versions.
A1111 version is version: [v1.2.1]• python: 3.9.12 • torch: 1.13.1+cu117 • xformers: N/A • gradio: 3.29.0
Controlnet version is v1.1.171.
My command line argument to start is
bash webui.sh --api
@lllyasviel Is this works for you, request you to add this in the wiki API.
Here is an example that works with img2img
"alwayson_scripts": {
"controlnet": {
"args": [
{
'enabled': True,
'module': "tile_resample",
'model': "control_v11f1e_sd15_tile [a371b31b]",
'weight': 1,
'resize_mode': 1,
'low_vram': False,
'processor_res': 512,
'threshold_a': 0,
'threshold_b': 1,
'guidance_start': 0,
'guidance_end': 1,
'control_mode': 1,
'pixel_perfect': True
},
{
'enabled': False,
'module': "reference_only",
'weight': 1,
'resize_mode': 1,
'low_vram': False,
'processor_res': 512,
'threshold_a': 0,
'threshold_b': 1,
'guidance_start': 0,
'guidance_end': 1,
'control_mode': 0,
'pixel_perfect': True
}
]
},
not working too. cn d831043cb81e97724ccf9f071da391d479440a77 python:3.10.6 • torch: 2.0.0+cu118 • xformers: 0.0.18 • gradio: 3.23.0 • commit: [22bcc7be]
txt2img works well via api. just the controlnet param not working( not working means there was no controlnet effect in the generated images)
is there anywhere i can check the log / meta info of the images generated by api? in infinite browser there are only images generated by webui.
----updated: it works after updating webui to 89f9faa63388756314e8a1d96cf86bf5e0663045 spent a whole day on this, crying.
So, I just started with a clean install of a1111, installed controlnet, ran the controlnet test script provided by @lllyasviel.
It looks like it's at least trying to use controlnet, because I get this error now:
Error running process: D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\a1111install\stable-diffusion-webui\modules\scripts.py", line 418, in process script.process(p, *script_args) File "D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1010, in process image = image_dict_from_any(unit.image) File "D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 153, in image_dict_from_any image = {'image': image[0], 'mask': image[1]} IndexError: list index out of range
EDIT: EURIKA!! It works now. I just realized i still was using old code that was using encoded_image as a [list]. Once I changed it to a string, the API worked great! Thank you @lllyasviel and everyone else for your help!
I believe the issue was that I was in fact using an old version of A1111. I was using the autoinstaller/launcher and thought that the "update webui" checkbox updated a1111 from the repository, but apparently that is wrong. Noob mistake on my part!
So, I just started with a clean install of a1111, installed controlnet, ran the controlnet test script provided by @lllyasviel.
It looks like it's at least trying to use controlnet, because I get this error now:
Error running process: D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\a1111install\stable-diffusion-webui\modules\scripts.py", line 418, in process script.process(p, *script_args) File "D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1010, in process image = image_dict_from_any(unit.image) File "D:\a1111install\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 153, in image_dict_from_any image = {'image': image[0], 'mask': image[1]} IndexError: list index out of rangeEDIT: EURIKA!! It works now. I just realized i still was using old code that was using encoded_image as a [list]. Once I changed it to a string, the API worked great! Thank you @lllyasviel and everyone else for your help!
I believe the issue was that I was in fact using an old version of A1111. I was using the autoinstaller/launcher and thought that the "update webui" checkbox updated a1111 from the repository, but apparently that is wrong. Noob mistake on my part!
I have encountered the same problem as you, but I don't quite understand what you said [I just realized i still was using old code that was using encoded_image as a [list]], can you explain it in more detail, please
