How to train on top of ControlNet model?

[whydna]

Trying to run the fill50K tutorial, but instead of training from scratch I want to train on top of an existing ControlNet model (say Canny). How would I go about doing that?

[santhosh97]

Hey @whydna , you can download the canny checkpoint here: https://huggingface.co/lllyasviel/ControlNet. And then in your training script, https://github.com/lllyasviel/ControlNet/blob/16ea3b5379c1e78a4bc8e3fc9cae8d65c42511b1/tutorial_train.py#L11, you would just change resume_path to the location of the downloaded checkpoint.

[geroldmeisinger]

I'm using the Huggingface diffusers control net training script https://huggingface.co/docs/diffusers/training/controlnet it outputs checkpoints every 500 steps by default with all the intermediate data needed to resume the training

controlnet/
	config.json
	diffusion_pytorch_model.safetensors
optimizer.bin
random_states_0.pkl
scaler.pt
scheduler.bin

I don't know whether or not this is also the case for the original training script, or if this data is available for the original control nets. but this is what you would need.