Dave Lage

Yeah I have had this issue myself as well but haven't identified why it happens sometimes. Probably need to identify why the alpha_cumprod is on non-CPU device and then this...

The issue is https://github.com/kohya-ss/sd-scripts/blob/7c075a9c8d234fccf8e0d66b9538a0b17bf4b13f/library/train_util.py#L5977-L6011 noise_scheduler.add_noise() moves the alphas_cumprod to the GPU And you can see it how it is working in the DDIMScheduler https://github.com/huggingface/diffusers/blob/a4df8dbc40e170ff828f8d8f79c2c861c9f1748d/src/diffusers/schedulers/scheduling_ddim.py#L474-L498 So this issue is for sd...

Issue right now is we are caching the neighbors into a file but saving it into the output_dir. this means each run we make a new file. We could: -...

> @rockerBOO i plan to test this > > this is only for flux lora? Yes only Flux LoRA for the moment

- Updated loss modification functions to take image_size for dynamically shifting timesteps (timestep_sampling == "flux_shift"). - Added tests for loss modification in custom train functions - Updated FlowMatchEulerDiscreteScheduler - Added...

8GB is possible but on 2000 series nvidia it doesn't support BF16, which Flux is stable in for training. Training in FP16 causes the loss to immediately NaN. You can...

I've been doing some further poking and got the VRAM down to 5.8GB by maximizing the block swaps (beyond whats configurable right now) and on my CPU seems to have...

Just tried it and its recording for me 

what commit are you currently on of sd-scripts?

I am looking into this issue but if anyone having this issue can confirm any wandb warnings in their terminal/command/bat output?