[Bug]: API /controlnet/txt2img route no longer working

[andrelevi]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

Hello,

I'm trying to use the txt2img API shown in the examples: https://github.com/Mikubill/sd-webui-controlnet/blob/main/example/api_txt2img.ipynb

When installing the latest sd-webui-controlnet main branch (commit 3a1c5b9), the /controlnet/txt2img route no longer works, and cannot be found in /docs anymore.

It used to work, and then I updated this extension to get access to the recently merged [multiple control units PR](https://github.com/Mikubill/sd-webui-controlnet/pull/384). After updating to the latest, the txt2img route is no longer working.

I also tried install a completely fresh stable-diffusion-webui repo, and then installing the sd-webui-control extension fresh, but I still have this issue.

I'm eager to try the new multiple control units via API feature!

Steps to reproduce the problem

1. Install the most recent main branch`
2. Look at /docs to see no txt2img branch

What should have happened?

/controlnet/txt2img should work like before.

Commit where the problem happens

webui: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8 controlnet: 3a1c5b9

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--medvram --api --api-log --allow-code

Console logs

N/A

Additional information

No response

[ljleb]

Are you sure? If you scroll up a bit, they should be there. To reuse the code from /sdapi/v1/txt2img and /sdapi/v1/img2img routes and extend them, we had to monkey patch the modules.api.api.Api class, which relocated the /controlnet/*2img routes docs just under the last base webui routes.

[ljleb]

Maybe we should move everything into the hijack to make sure all /controlnet routes stick together in the docs. Alternatively, find a way to reuse the Api code without hijacking it.

[ljleb]

Have you tried installing the extension, then restarting your webui completely? I suspect this may be caused by the monkey patch not taking place.

This is what I see on my end:

[ljleb]

Wait, why is everything listed under / instead of /sdapi/v1 on your end? For me every route the ui defines before extensions is nested into /sdapi/v1.

[ljleb]

I just tried these steps:

1. clone a fresh webui repo into a new directory
2. go to extensions tab > available > click on "Load from:"
3. click on "Install" button for sd-webui-controlnet
4. go to installed tab > click on "Apply and restart UI"
5. open a new browser tab at http://localhost:7860/docs
6. see that the /controlnet/*2img routes are indeed displayed there

The script shared above works on my side: (the only line I changed is the controlnet_model param to use a diff model instead)

[ControlNet] warning: using deprecated 'controlnet_*' request params
[ControlNet] warning: consider using the 'control_units' request param instead
Loading model: diff_control_sd15_canny_fp16 [ea6e3b9c]
Loaded state_dict from [E:\sd\models\ControlNet\diff_control_sd15_canny_fp16.safetensors]
Offset cloned: 298 values
ControlNet model diff_control_sd15_canny_fp16 [ea6e3b9c] loaded.
Loading preprocessor: canny

I am not sure what is going on, because the hijack is happening on my end without the need to restart anything. Are you sure you shared all the command line arguments used to start the webui?

I don't think this could be caused by a browser extension, but just to be sure could you also make sure you don't have any browser extension enabled when navigating to /docs?

[Fictiverse]

They changed completly the json structure, and the base64 image format is also not the same as the default base64 format used on original txt2img and img2img vanilla API. You need to remove the "data:image/png;base64," prefix

[andrelevi]

Hi, thank you for your help!

Wait, why is everything listed under / instead of /sdapi/v1 on your end? For me every route the ui defines before extensions is nested into /sdapi/v1.

Really? Do I have an older version of webui? I literally cloned down the latest via: git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

I'm on Windows, using Chrome.

Here is what I see for routes:

[andrelevi]

They changed completly the json structure, and the base64 image format is also not the same as the default base64 format used on original txt2img and img2img vanilla API. You need to remove the "data:image/png;base64," prefix

So after I get my controlnet routes working again...

Does that mean that I cannot simply pass the base64 string after opening an image in python like: b64img = readImage("/root/workspace/nahida/0e17302b9bfa15402f783c29c0d1d34f.jpg")? I need remove the "data:image/png;base64," prefix in the generated b64img string?

Sorry just want to confirm, since back when I did have working controlnet routes, I still couldn't get the txt2img API working when passing in a base64 image.

[Fictiverse]

On the previous version of the API it is how it worked, not sur for now, i didnt found how to make it work again since the new update.

Try to chek here https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/api.py

[andrelevi]

On the previous version of the API it is how it worked, not sur for now, i didnt found how to make it work again since the new update.

Try to chek here https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/api.py

Gotcha, thanks for this info.

Btw, are you able to see the controlnet/txt2img route with the new update?

[ljleb]

They changed completly the json structure, and the base64 image format is also not the same as the default base64 format used on original txt2img and img2img vanilla API. You need to remove the "data:image/png;base64," prefix

The function that decodes the base64 images is this one: https://github.com/Mikubill/sd-webui-controlnet/blob/2ce17c0a2bb661ca186764c3b5fe79997cb267e6/scripts/api.py#L21-L22

It uses the webui's function, which does strip the prefix: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8/modules/api/api.py#L54-L61

It should therefore work if the base64 images are prefixed with "data:image/png;base64,".

Note that the previous json structure that was using controlnet_* fields at the root of the object should still work, however it is not displayed in the docs anymore because using a list of dicts makes more sense as we can use multiple control units now. You will get a warning in console if using at least one of the deprecated fields.

[ljleb]

Really? Do I have an older version of webui? I literally cloned down the latest via: git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

I'm on Windows, using Chrome.

I used chrome when I failed to reproduce earlier. Can you share the output of git rev-parse HEAD with a console inside your /sd-webui dir as well as inside /sd-webui/extensions/sd-webui-controlnet? Is your clone clean, i.e. any error reported by git status on any of the 2 repos?

[ljleb]

Alright I just realized. All of your /sdapi/v1 routes are gone. It's not the prefix that's missing, but every route defined under it. I think the hijack is doing something your python interpreter does not like. I'll try to find a better way to extend the original api.Api class.

[andrelevi]

My stable-diffusion-webui repo has a clean status, aside from the altered args: +export COMMANDLINE_ARGS="--medvram --api --api-log --allow-code"

andre@MSI   base  D:\..\..\stable-diffusion-webui   master ≣ +2 ~1 -0 !  git rev-parse HEAD

0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8

sd-webui-controlnet:

andre@MSI   base  D:\..\..\stable-diffusion-webui\..\sd-webui-controlnet   main ≣  pwd                                                                               
Path
----
D:\Work\webui-test\stable-diffusion-webui\extensions\sd-webui-controlnet


andre@MSI   base  D:\..\..\stable-diffusion-webui\..\sd-webui-controlnet   main ≣  git status

On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

andre@MSI   base  D:\..\..\stable-diffusion-webui\..\sd-webui-controlnet   main ≣  git rev-parse HEAD
3a1c5b9f3ce050578e126575cca0ff29d29accab

[andrelevi]

Alright I just realized. All of your /sdapi/v1 routes are gone. It's not the prefix that's missing, but every route defined under it. I think the hijack is doing something your python interpreter does not like. I'll try to find a better way to extend the original api.Api class.

Ah, interesting! My python version is:

andre@MSI  D:\..\..\stable-diffusion-webui   master ≣ +2 ~1 -0 !  python --version
Python 3.10.10

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]

[ljleb]

Thanks for the info.

Are /sdapi/v1 routes available if you disable the extension? what if you uninstall it and restart the webui completely?

If the extension is causing the routes to disappear, I tried an alternative way to hijack api.Api that might be less intrusive. In this case, could you try this: (in the directory of the controlnet extension, should be at sd-webui/extensions/sd-webui-controlnet)

git remote add ljleb https://github.com/ljleb/sd-webui-controlnet.git
git pull ljleb alternative-api-hijack

And then try to start the webui again. It will add an alternative remote pointing to my fork and then apply the diff.

Once you're done testing and want to revert these changes, in case you don't know how: (apologies if you do, I just try to be helpful 😅):

git reset --hard origin
git remote remove ljleb

[andrelevi]

Are /sdapi/v1 routes available if you disable the extension? what if you uninstall it and restart the webui completely?

Ahh, so I made a fresh clone of: https://github.com/AUTOMATIC1111/stable-diffusion-webui

And I don't see any of the /sdapi/v1 routes at /docs!

Sorry, it seems like the extension is not at fault then.

Do you have an idea on why I might be missing those routes? If they are for sure supposed to be there, I will try opening an issue at: https://github.com/AUTOMATIC1111/stable-diffusion-webui

[ljleb]

Ah nice, glad we narrowed it down.

Do you have an idea on why I might be missing those routes? If they are for sure supposed to be there, I will try opening an issue at: https://github.com/AUTOMATIC1111/stable-diffusion-webui

Unfortunately, except maybe for environment variables, I don't have an idea. I think opening an issue in the webui repo might be the way to go.

I'll close this as it seems to be unrelated to controlnet, but feel free to reopen if you have more info and it turns out the extension really was at fault.

[Bruce-shuai]

i am a mac user, and i solve this problem by add --api in webui-macos-env.sh file's COMMANDLINE_ARGS fields。like this: /** webui-macos-env.sh **/ export COMMANDLINE_ARGS="--api --skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate"