Fine-tuning stable diffusion

[xuanxu92]

hi, thanks for this excellent work. Could you please share the code for finetuning stable diffusion to get the pretrained stable diffusion checkpoint? Thanks!

[IceClear]

Hi～our approach does not need to finetune the SD part. We directly use the official checkpoint. You can find the link in readme.

[xuanxu92]

Thanks for the reply! I see in your paper in experiment detail part that mention fine tune the diffusion model of stableSR, so that means fine tune the SFT part, right?

Thanks for your time and help!

Our time-aware encoder is similar to the contracting path of the denoising U-Net in Stable Diffusion but is much more lightweight (∼105M, including SFT layers). SFT layers are inserted in each residual block of Stable Diffusion for effec- tive control. We finetune the diffusion model of StableSR for 117 epochs with a batch size of 192, and the prompt is fixed as null.

[IceClear]

Thanks for the reply! I see in your paper in experiment detail part that mention fine tune the diffusion model of stableSR, so that means fine tune the SFT part, right?

Thanks for your time and help!

Our time-aware encoder is similar to the contracting path of the denoising U-Net in Stable Diffusion but is much more lightweight (∼105M, including SFT layers). SFT layers are inserted in each residual block of Stable Diffusion for effec- tive control. We finetune the diffusion model of StableSR for 117 epochs with a batch size of 192, and the prompt is fixed as null.

Yes.

[xiliu8006]

Hi, does it mean the pretrained SFT module include the weights of Stable diffusion? So we do not need to load the SD weights from huggingface? Because I haven't seen any code to download the stable diffusion weights.