AI-Render
AI-Render copied to clipboard
benrugg
ControlNet funny result
Describe the bug
Using a ControlNet Rig in Blender + AI render with ControlNet more recent control_v11p_sd15_openpose.pth (https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main) I get this funny result:
(A future statue erected to AI maybe?)
My settings:
Any idea what I missed?
To reproduce
Install openpose from https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main add a openpose rig + mesh setup as above F12
Error log
none, but ControlNet openpose should work
Environment
- Blender version (upper right corner of splash screen): 3.3.7
- AI Render version (find in Preferences > Add-ons): 0.7.9
- Operating system (Windows/Mac/Linux): Ubuntu 22.04
Screenshots/video
No response
Additional information
No response
Please @steve-salmond could you please explain how you get some ControlNet result? I'm pretty sure I'm doing something wrong here :smile:
Hmm, that's no good. Thanks for posting this. I'm not sure what's going wrong. While developing this feature, I tested with the depth model and one of the architectural ones, but I didn't actually test with openpose.
If I had to guess, I think there's a possibility that the preprocessor is the problem. I think the point of the preprocessor is to take a regular image (like one of a person) and then turn it into an image for the model to use (like a pose).
So maybe try without any preprocessor.
(I am traveling right now, without computer, so I won't be able to help much until later in the week. But feel free to send me more info on whatever you try!)
AFAIK, prepocessor to none wouldn't treat the skeleton as a pose.
I actually think that's exactly the way ControlNet works. With a preprocessor, it converts a regular image into a skeleton image. Without it, it uses the image input as the skeleton image.
I could be wrong - I haven't tested this with the openpose model. But I think this would be the best next step for you to try.
Ok, I see what you mean.
Nope, it keeps on using the pose as image, not as pose:
I guess my process is wrong. Are you successful using a pose?
You're right about the preprocessor, in webui @benrugg , it is not needed.
t works only in webui, if I select Control Type "All" or "openpose":
But if I select Control Type as "canny":
So I guess, we're missing a selection of priority_order in your code?
Ok, yeah, there must be a different way that Automatic1111's API is expecting the input. I will check it out and try to fix it when I have a chance. I am in the middle of a launch on a different (unrelated) product right now, so it might be a few weeks before I can dedicate time to this.
Hey, sorry, I can't give an ETA. Like I mentioned, I'm head down on a different project right now, so it will probably be a while. If you can find anyone else who can experiment with this, I'd be happy to review a PR.
Sure, but I need some little guidance first :smile:
Do you also think we miss priority_order ?
Quickly, do you have hints how to add it in the addon UI?
Thanks
I think it will likely take some experimentation to figure out what data and settings need to be passed to Automatic1111's API. My intuition is that it's not priority_order, but it's possible. I think someone will need to dive into their API and my python code.
I can try to dive in, I'd just need to know how to add a setting like priority_order in the UI.
I think for better or worse, the UI would probably be the last step in the process. Most of the work would be in python. Blender has a lot of guides, but it takes a lot to dive into it unless you already know how to code.