fast-stable-diffusion icon indicating copy to clipboard operation
fast-stable-diffusion copied to clipboard

Feature Request - Saving intermediary checkpoints

Open DominikDoom opened this issue 2 years ago • 4 comments

I don't know if it's possible in this architecture, but some of the popular non-diffuser implementations allow you to save an intermediary checkpoint every X steps. This would be a really great feature for here as well, since it would make it easier to compare different step counts and also mitigate the risk of losing the model due to Google disconnecting the instance on the free tier.

This would also allow you to set the steps really high and check if the quality is good enough already for each intermediary, cancelling when it's enough. Even better would be a combination with logged test images generated at a similar step interval, but not sure if it would add too much overhead here since all previewing was handed off to the external webuis.

DominikDoom avatar Oct 04 '22 16:10 DominikDoom

That would require tapping into the trainer script to download the model from the VRAM at a chosen time, you can request that in the main repo of the dreambooth script

TheLastBen avatar Oct 05 '22 02:10 TheLastBen

The colab has been updated to perform in-session inference

TheLastBen avatar Oct 05 '22 02:10 TheLastBen

The colab has been updated to perform in-session inference

Do you mean in-session image generation? Or saving/exporting ckpt at certain # of steps

timdesrochers avatar Oct 08 '22 03:10 timdesrochers

image generation

TheLastBen avatar Oct 08 '22 06:10 TheLastBen

Closing this since it's resolved now. (See #206 which was implemented in https://github.com/TheLastBen/diffusers/commit/0b77c4e56bf377d424c7ed87d6cb1804469b0217).

DominikDoom avatar Nov 01 '22 14:11 DominikDoom