Stepfunction

Before anyone else tries this, it seems to break the guidance module that Ostris created. It seems some more work will be needed to explicitly exclude that from training.

> > My 3090 does 7.89 it's.. > > [@CodeAlexx](https://github.com/CodeAlexx) . Are you saying 7.89 seconds per iterations ? That sounds about right given it's all 1024 resolution images. I'm...

The samples look decent, but if you try to use it in Comfy, you'll find that guidance no longer works. The training process is likely training that part of the...

https://github.com/kohya-ss/sd-scripts/issues/1891#issuecomment-2608545488 If anyone wants to play with this, I've created a minimal working example here: https://github.com/stepfunction83/sd-scripts/tree/sd3 With this commit just brute forcing in the relevant code snippets from ai-toolkit: https://github.com/kohya-ss/sd-scripts/commit/b203e318774e9a8a6d64a2aea9173cf259dc70ea...

Make sure to pass the --bypass_flux_guidance parameter with the latest commit, and yes, you can just replace the respective files with the ones from the forked version.

Yep, let me know how your experience goes. I'll submit a PR once I get it in a slightly better state.

That's currently the only way I've tested it, so ensuring it works for LoRA too is still needed. I'll probably try that tomorrow after this finetune run finishes. On Wed,...

Created a PR to add the functionality to sd-scripts: https://github.com/kohya-ss/sd-scripts/pull/1893

From my experimentation with finetuning so far, I've found that lower learning rates are needed than with Flux Dev. 5e-6, Cosine, 5000 steps destroyed hands and general composition, but 1e-6,...

I think an LR even lower than 1e-6 may be better. Even with that, it trains quickly and reaches approximately the same place as a 5e-6 LR in 5000 steps,...