Do we need to do all those conversions for inpaining?

[geekyayush]

Hello Do we need to do all those conversions mentioned under "ControlNet + Anything-v3" for inpainting?

Also, in the inpainting guide, there are these lines

# we have downloaded models locally, you can also load from huggingface
# control_sd15_seg is converted from control_sd15_seg.safetensors using instructions above
pipe_control = StableDiffusionControlNetInpaintPipeline.from_pretrained("./diffusers/control_sd15_seg",torch_dtype=torch.float16).to('cuda')
pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained("./diffusers/stable-diffusion-inpainting",torch_dtype=torch.float16).to('cuda')

Can anyone help understand what does the 2nd line means? Also, for pipe_inpaint , do we pass the stable diffusion diffuser model path?

[haofanwang]

Thanks for your interest! @geekyayush

1. Yes, you should strictly follow our instructions.
2. pipe_inpaint is a inpainting model based on stable diffusion, we use runwayml/stable-diffusion-inpainting. We cannot directly load a stable diffusion model such as runwayml/stable-diffusion-v1-5, although they are both based on stable-diffusion-1.5, their input channels are different.

[UglyStupidHonest]

Hey just following up here ! This might be a newbie misconception but if we replace the unet here do we not loose the custom model in this case the Anything v3 . Or is it really just replacing the inpainting channels ?

[geekyayush]

Thanks @haofanwang !

I have another question regarding this. If I want to use an SD + Dreambooth trained inpainting fine-tuned model, then, will it work this line?

pipe_control = StableDiffusionControlNetInpaintPipeline.from_pretrained("./diffusers/control_sd15_seg",torch_dtype=torch.float16).to('cuda')
pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained("./diffusers/my-dreambooh-inpaint-model",torch_dtype=torch.float16).to('cuda')

Here, for the pipe_control, I am using the same control_sd15_seg model and for pipe_inpaint , I am using my custom trained model.

Thanks!

[haofanwang]

Let me answer all you guys concerns here.

@UglyStupidHonest You are right, for now, if you want to equip ControlNet with inpainting ability, you have to replace the whole base model, which means that you cannot use anything-v3 here. I did try to only replace the input layer and keep all other layers in anything-v3, but it works bad.

@geekyayush If you inpainting model has the exact same layers as stable-diffusion-1.5, then it should work. You can just take ControlNet as a pluggable module that can insert into all stable-diffusion-1.5 based models.

[hyperia-elliot]

Is StableDiffusionControlNetInpaintPipeline currently operable? Trying the sample code in this repo with the provided input images, segmentation map, and specified models, gives the following result in my environment:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ │ │ /ingest/ImageDiffuserService/client/inpaint_proto.py:31 in │ │ │ │ 28 # the segmentation result is generated from https://huggingface.co/spaces/hysts/ControlN │ │ 29 control_image = load_image('segmap.png') │ │ 30 │ │ ❱ 31 image = pipe_control(prompt="Face of a yellow cat, high resolution, sitting on a park be │ │ 32 │ │ │ │ │ negative_prompt="lowres, bad anatomy, worst quality, low quality", │ │ 33 │ │ │ │ │ controlnet_hint=control_image, │ │ 34 │ │ │ │ │ image=image, │ │ /home/devuser/anaconda3/envs/pytorch-env/lib/python3.8/site-packages/torch/autograd/grad_mode.py: │ │ 27 in decorate_context │ │ │ │ 24 │ │ @functools.wraps(func) │ │ 25 │ │ def decorate_context(*args, **kwargs): │ │ 26 │ │ │ with self.clone(): │ │ ❱ 27 │ │ │ │ return func(*args, **kwargs) │ │ 28 │ │ return cast(F, decorate_context) │ │ 29 │ │ │ 30 │ def _wrap_generator(self, func): │ │ │ │ /home/devuser/anaconda3/envs/pytorch-env/lib/python3.8/site-packages/diffusers/pipelines/stable_d │ │ iffusion/pipeline_stable_diffusion_controlnet_inpaint.py:793 in call │ │ │ │ 790 │ │ │ │ │ │ 791 │ │ │ │ if controlnet_hint is not None: │ │ 792 │ │ │ │ │ # ControlNet predict the noise residual │ │ ❱ 793 │ │ │ │ │ control = self.controlnet( │ │ 794 │ │ │ │ │ │ latent_model_input, t, encoder_hidden_states=prompt_embeds, cont │ │ 795 │ │ │ │ │ ) │ │ 796 │ │ │ │ │ control = [item for item in control] │ │ │ │ /home/devuser/anaconda3/envs/pytorch-env/lib/python3.8/site-packages/torch/nn/modules/module.py:1 │ │ 130 in _call_impl │ │ │ │ 1127 │ │ # this function, and just call forward. │ │ 1128 │ │ if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o │ │ 1129 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │ │ ❱ 1130 │ │ │ return forward_call(*input, **kwargs) │ │ 1131 │ │ # Do not call functions when jit is used │ │ 1132 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │ │ 1133 │ │ if self._backward_hooks or _global_backward_hooks: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: forward() got an unexpected keyword argument 'controlnet_hint'