Conversion to HF ControlNet ckpt

[CesarERamosMedina]

Note: this is not an issue, but I couldn't find a better place to submit this. I love this! In the paper, you mention that for Stable Diffusion, you trained using ControlNet. However, I also see that instead of using the ControlNet hacked version of DDIM, you are using DDIM with Spacial CFG. I was wondering, do you think that the HuggingFace diffusers convert ControlNet to Diffusers ckpt script work out of the box for PAIR-Diffusion, given that difference?

I am still finding my way around latent diffusion models, but if the above is not the case, I would love to be able to help or follow any effort in converting the existing code into a HuggingFace pipeline.

[vidit98]

Hi @CesarERamosMedina , I am glad that you liked the paper. Another difference other than spatial CFG is that we do not use a network for self.input_hint_block instead we simply interpolate the VGG features to 64x64. I am not sure if diffusers have this option in Controlnet. Regarding, spatial CFG I think we can get around that, it is an extra controllability feature of PAIR Diffusion but simple DDIM should work as well with reduced controllability.

It would be really helpful if you can help in integrating it into the HuggingFace pipeline. If you have any doubts please let me know

[CesarERamosMedina]

Thank you so much for the fast reply! Will let you know of any other questions