How to use controlNet with pretrained LoRA?

[zengqixun12]

I want to combine the pretrained LORA_model and pretrained controlnet_mdoel like "[control_sd15_depth.pth]",and export new model.

How to realize this function?

[JamesSand]

I have the same problem with you, is there any solution?

[zengqixun12]

I have the same problem with you, is there any solution?

I've solved this problem, and I combined Lora + Train + ControlNet + change basemodel, export the final result and the model without Stable-Diffusion-WebUI (only use the command line)

[zengqixun12]

I have the same problem with you, is there any solution?

I will release the code in few days ;)

[JamesSand]

I have the same problem with you, is there any solution?

I will release the code in few days ;)

thanks a lot. I am also trying to solve this issue. May I ask should we apply lora on the trainable copy of controlnet?

[zengqixun12]

Not exactly, Lora is only effective on the main network and is independent from ControlNet

[SnowdenLee]

Hi @zengqixun12, may I ask do you still plan to release the code? Thank you :)

[gwyong]

@zengqixun12 Hello, do you still have plan to release the code? Thanks,

[jiachen0212]

I have the same problem with you, is there any solution?

I will release the code in few days ;)

thanks a lot. I am also trying to solve this issue. May I ask should we apply lora on the trainable copy of controlnet?

You can try: first merge the SD model of base and the parameters of lora, similar to the merger of con parameters and bn parameters. Then train your own data according to the document of train.md. I followed this plan and it worked.