lllyasviel
img4img(with controlnet) api error , please help me.
*** Error running process: D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\webui_forge\webui\modules\scripts.py", line 844, in process script.process(p, *script_args) File "D:\webui_forge\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 554, in process self.process_unit_after_click_generate(p, unit, params, *args, **kwargs) File "D:\webui_forge\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 304, in process_unit_after_click_generate input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 233, in get_input_data image = self.try_crop_image_with_a1111_mask(p, unit, image, resize_mode, preprocessor) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 126, in try_crop_image_with_a1111_mask preprocessor.corp_image_with_a1111_mask_when_in_img2img_inpaint_tab AttributeError: 'NoneType' object has no attribute 'corp_image_with_a1111_mask_when_in_img2img_inpaint_tab'
[Unload] Trying to free 1231.42 MB for cuda:0 with 0 models keep loaded ... Done. [Memory Management] Target: IntegratedAutoencoderKL, Free GPU: 13549.72 MB, Model Require: 159.56 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 12366.16 MB, All loaded to GPU. Moving model(s) has taken 0.24 seconds [Unload] Trying to free 1329.14 MB for cuda:0 with 0 models keep loaded ... Current free memory is 13377.64 MB ... Done. [Memory Management] Target: JointTextEncoder, Free GPU: 13377.64 MB, Model Require: 234.72 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 12118.92 MB, All loaded to GPU. Moving model(s) has taken 1.10 seconds [Textual Inversion] Used Embedding [EasyNegative] in CLIP of [clip_l] [Unload] Trying to free 1024.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 13035.71 MB ... Done. *** Error running process_before_every_sampling: D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\webui_forge\webui\modules\scripts.py", line 892, in process_before_every_sampling script.process_before_every_sampling(p, *script_args, **kwargs) File "D:\webui_forge\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in process_before_every_sampling self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs) KeyError: 0
[Unload] Trying to free 1556.81 MB for cuda:0 with 0 models keep loaded ... Current free memory is 13035.42 MB ... Done. [Memory Management] Target: KModel, Free GPU: 13035.42 MB, Model Require: 0.00 MB, Previously Loaded: 1639.41 MB, Inference Require: 1024.00 MB, Remaining: 12011.42 MB, All loaded to GPU. Moving model(s) has taken 0.02 seconds 100%|██████████████████████████████████████████████████████████████████████████████████| 16/16 [00:02<00:00, 5.56it/s] [Unload] Trying to free 1089.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 13027.23 MB ... Done. *** Error running postprocess_batch_list: D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\webui_forge\webui\modules\scripts.py", line 916, in postprocess_batch_list script.postprocess_batch_list(p, pp, *script_args, **kwargs) File "D:\webui_forge\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "D:\webui_forge\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 567, in postprocess_batch_list self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs) KeyError: 0
You get this error when trying to use the API? Can you share your JSON payload?
You get this error when trying to use the API? Can you share your JSON payload?
Thanks for your reply! here is the json and api call
payload = {'init_images': [encoded_image], 'prompt': 'cartoon', 'negative_prompt': 'EasyNegative, nsfw', 'denoising_strength': 0.75, 'width': 512, 'height': 512, 'cfg_scale': 17, 'sampler_name': 'DPM++ 2M Karras', 'steps': 20, 'seed': 1, 'alwayson_scripts': {'controlnet': {'args': [ { 'enabled': True, 'input_image': encoded_image, 'module': None, 'model': 'control_v11e_sd15_ip2p_fp16 [fabb3f7d]', 'weight': 1, 'resize_mode': 0, 'lowvram': False, 'processor_res': 64, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 0, 'pixel_perfect': False }, { 'enabled': True, 'input_image': encoded_image, 'module': 'canny', 'model': 'control_v11p_sd15_canny_fp16 [b18e0966]', 'weight': 0.8, 'resize_mode': 0, 'lowvram': False, 'processor_res': 512, 'threshold_a': 100, 'threshold_b': 200, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 0, 'pixel_perfect': False }, { 'enabled': True, 'input_image': encoded_image2, 'module': 'shuffle', 'model': 'control_v11e_sd15_shuffle_fp16 [04a71f87]', 'weight': 1, 'resize_mode': 0, 'lowvram': False, 'processor_res': 512, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 2, 'pixel_perfect': False } ]} } }
response = requests.post(url=f'{url}/sdapi/v1/img2img', json=payload)
The above json runs fine under 1111 webui. Is it because the APIs of 1111 webui and webui forge are incompatible?
They should be roughly identical, however, it may differ for the controlnet settings because Forge's built-in controlnet is a little different from the controlnet extension.
This worked for me (Just txt2img this time):
POST http://127.0.0.1:7860/sdapi/v1/txt2img HTTP/1.1
Content-Type: application/json
Accept: application/json
{
"prompt": "cartoon",
"negative_prompt": "EasyNegative, nsfw",
"width": 1024,
"height": 1024,
"cfg_scale": 5,
"sampler_name": "Euler a",
"scheduler": "Align Your Steps",
"steps": 20,
"seed": 1,
"alwayson_scripts": {
"controlnet": {
"args": [
{
"enabled": true,
"image": "base64_image",
"module": "canny",
"model": "xinsir-sdxl-canny [ab7dc06d]",
"weight": 0.8,
"resize_mode": 0,
"lowvram": false,
"processor_res": 512,
"threshold_a": 100,
"threshold_b": 200,
"guidance_start": 0,
"guidance_end": 1,
"control_mode": 0,
"pixel_perfect": true
}
]
}
}
}
According to the controlnet API docs (for the extension), instead of input_image, it would just be image for encoded image field. Additionally, be careful about how you supply sampler/scheduler names to the API. DPM++ 2M Karras is not recognized.
In case you are not aware, you can use http://127.0.0.1:7860/docs to view the API info and checkout the allowed parameters.
Thanks for your reply! I seem to have found the reason. If I remove the first controlnet (control_v11e_sd15_ip2p_fp16 [fabb3f7d]) in the code below, there will be no error when running. Is it because forge API does not support ip2p? However, when I run it in UI, ip2p can work.
{'init_images': [encoded_image], 'prompt': 'cartoon', 'negative_prompt': 'EasyNegative, nsfw', 'denoising_strength': 0.75, 'width': 512, 'height': 512, 'cfg_scale': 17, 'sampler_name': 'Euler a', 'scheduler': 'Align Your Steps', 'steps': 20, 'seed': 1, 'alwayson_scripts': {'controlnet': {'args': [ { 'enabled': True, 'image': encoded_image, 'module': None, 'model': 'control_v11e_sd15_ip2p_fp16 [fabb3f7d]', 'weight': 1, 'resize_mode': 0, 'lowvram': False, 'processor_res': 64, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 0, 'pixel_perfect': False }, { 'enabled': True, 'image': encoded_image, 'module': 'canny', 'model': 'control_v11p_sd15_canny_fp16 [b18e0966]', 'weight': 0.8, 'resize_mode': 0, 'lowvram': False, 'processor_res': 512, 'threshold_a': 100, 'threshold_b': 200, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 0, 'pixel_perfect': False }, { 'enabled': True, 'image': encoded_image2, 'module': 'shuffle', 'model': 'control_v11e_sd15_shuffle_fp16 [04a71f87]', 'weight': 1, 'resize_mode': 0, 'lowvram': False, 'processor_res': 512, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 2, 'pixel_perfect': False } ]} } }
What does the PNG info say of an image generated with pix2pix from the UI? That might give us a clue on how to setup the JSON correcctly.
Thank you for your reply. Here is the information of UI
cartoon Negative prompt: EasyNegative, nsfw Steps: 20, Sampler: Euler a, Schedule type: Align Your Steps, CFG scale: 7, Seed: 1597812254, Size: 512x512, Model hash: e9476a1372, Model: v1-5-pruned-emaonly-fp16, Denoising strength: 0.75, ControlNet 0: "Module: None, Model: control_v11e_sd15_ip2p_fp16 [fabb3f7d], Weight: 1, Resize Mode: Crop and Resize, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both", TI: EasyNegative, Version: f2.0.1v1.10.1-previous-659-gc055f2d4
What does the PNG info say of an image generated with pix2pix from the UI? That might give us a clue on how to setup the JSON correcctly.
Dear Mr. MisterChief95 I'm very sorry to bother you often. I'm still troubled by this issue. Do you think it's a bug in the Forge API? Or is it my improper usage method? Looking forward to your reply, thank you very much!
Hi @xiaoxuwei12345 , I took another look. I was able to get it working in both txt2img and img2img. Notice that i dont have all the same parameters for the ip2p cnet. txt2img is the same, just different API endpoint and you will not include init_images.
POST http://127.0.0.1:7860/sdapi/v1/img2img HTTP/1.1
Content-Type: application/json
Accept: application/json
{
"init_images": ["encoded image"],
"prompt": "person underwater, swimming",
"checkpoint": "asyncsMIXILLUSTRIOUS_v71",
"negative_prompt": "nsfw",
"width": 512,
"height": 768,
"cfg_scale": 7,
"sampler_name": "DPM++ SDE",
"scheduler": "Karras",
"steps": 20,
"seed": 1,
"override_settings": {
"CLIP_stop_at_last_layers": 2
},
"alwayson_scripts": {
"controlnet": {
"args": [
{
"enabled": true,
"image": "encoded image",
"model": "control_v11e_sd15_ip2p_fp16 [fabb3f7d]",
"weight": 0.8,
"resize_mode": 0,
"lowvram": false,
"guidance_start": 0,
"guidance_end": 1,
"control_mode": 0,
"pixel_perfect": true
}
]
}
}
}
For your specific issue, remove this line from the ip2p cnet:
'module': None,
This causes the KeyError you were seeing.
Thank you very much! Yes, I also discovered yesterday that the reason is caused by this' module ': None. Because I added the print log in the webui_forge \ webui \ extensions builtin \ sd-forge_comtrol net \ scripts \ control net. py file yesterday, I found that when running in the UI, the value of preprocessor is< modules_forge.supported_preprocessor.PreprocessorNone object at 0x0000015DD1F11090>。It seems that A1111 and Forge are somewhat different. In A1111, 'module' can be set to None。
Also, I have a question that I don't quite understand. In A1111 controlnet, I set "input_image" : encoded_image, and it can run normally. In Forge, do I have to change it to "image" : encoded_image? Is this the difference between A1111 and Forge?
Please give me more guidance, thank you very much!
I am glad you got it working!
It appears that even in A1111, the CNet extension uses image as the input field, both in JSON and in the Python code itself:
https://github.com/Mikubill/sd-webui-controlnet/wiki/API#integrating-sdapiv12img
As for the module field being None, I would agree they are different here. That said, it is likely easier to simply not populate the field for pix2pix models. That's the only model i've seen that doesnt have its own module.
Thank you very much! I searched on Google using keywords such as "alwayson_scripts controlnet args input_image" and found that many examples used input_image, such as the following article:
https://randombits.dev/articles/stable-diffusion/controlnet-api
And I have been using the input_image parameter in A1111 and it has been running normally. I don't want to delve deeper into the specific reasons, since both you and the official documentation suggest using image (not input_image), then I will use image.
Thank you again!
Dear Mr. MisterChief95 Very, very sorry to bother you again. I seem to have discovered a bug in the Forge API. Perhaps I haven't mastered the correct usage method?
In a multi Controlnet API, if ip2p Controlnet is used, shuffle Controlnet will not work (there are no errors, but shuffle Controlnet has no effect). Based on my testing, I have only found that ip2p Controlnet affects shuffle Controlnet. Canny and other Controlnet can work properly with shuffle Controlnet.
This is a very strange phenomenon, in the A1111 API, ip2p Controlnet and shuffle Controlnet can work together and both have an effect.
By the way, just using ip2p Controlnet in the Forge API will render shuffle Controlnet ineffective. In Forge UI, ip2p Controlnet and shuffle Controlnet can work together and both have an effect.
The code is as follows:
{'init_images': [encoded_image], 'prompt': 'cartoon', 'negative_prompt': 'EasyNegative, nsfw', 'denoising_strength': 0.75, 'width': 512, 'height': 512, 'cfg_scale': 17, 'sampler_name': 'Euler a', 'scheduler': 'Align Your Steps', 'steps': 20, 'seed': 1, 'alwayson_scripts': {'controlnet': {'args': [ { 'enabled': True, 'image': encoded_image, 'model': 'control_v11e_sd15_ip2p_fp16 [fabb3f7d]', 'weight': 1, 'resize_mode': 0, 'lowvram': False, 'processor_res': 64, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 0, 'pixel_perfect': False }, { 'enabled': True, 'image': encoded_image2, 'module': 'shuffle', 'model': 'control_v11e_sd15_shuffle_fp16 [04a71f87]', 'weight': 2, 'resize_mode': 0, 'lowvram': False, 'processor_res': 512, 'threshold_a': 64, 'threshold_b': 64, 'guidance_start': 0, 'guidance_end': 1, 'control_mode': 2, 'pixel_perfect': False } ]} } }
response = requests.post(url=f'{url}/sdapi/v1/img2img', json=payload)