Is it possible to guide inpainting using controlnet?

[i-amgeek]

Model/Pipeline/Scheduler description

Hi, I am trying to inpaint a person in specific pose. Is it possible to use controlnet with this repo?

Open source status

• [ ] The model implementation is available.
• [ ] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[juxuan27]

Hi, @i-amgeek ! Thank you for your focus. I think it is theoretically sound to combine ControlNet and BrushNet together. But I haven't tried that personally. I may give it a try if I have the time. Moreover, we are welcome for any PR or discussion. Please tell us if you have tried that!

[loboere]

StableDiffusionBrushNetPipeline can load controlnet? or does it need to be modified?

[juxuan27]

I have retained the ControlNet functionality in this repo, but I am not sure whether it would work normally since I haven't tried that. If it works, please let me know! I will find time to try that maybe after a few days.

[jiechu520]

I have tried combining with controlnet，it works! need modified StableDiffusionBrushNetPipeline.

[juxuan27]

Hi, @jiechu520 ! Is that possible for you to pull your modification of StableDiffusionBrushNetPipeline to this repo and give us some examples? Thank you so much for your response!

[SlZeroth]

same!!! I want to use controlnet with brushnet. can you upload the code? I think it will be really nice quality with controlnet. because unother paper replace anything use controlnet

[YoucanBaby]

I have tried combining with controlnet，it works! need modified StableDiffusionBrushNetPipeline.

Hi, can you show the input and output? I tried combining it with controlnet and got unsatisfactory output.

[SlZeroth]

@YoucanBaby can you share your code? I will test and fix the issue

[juxuan27]

The node in comfyui has supported this. Issue closed.