PromeAI
PromeAI
sorry respond late, i just test a few minutes earlier. tried with 10 validation images and didn't need more CUDA memory. So can you provide more detailed config? especially `resolution`,...
really confused me, ive tried install same transformer\accelerate versions of yours and works good. may i ask that what's your total gpu number? are you testing training on multi gpu?...
maybe it is about precision. I guess accelerate try convert params to bf16 but fail and remain fp32? it is a guess it be device-related issue, need to test on...
Consider using these two options `--use_adafactor ` `--gradient_checkpointing` @Night1099 @Trav1slaflame
> Consider using these two options `--use_adafactor ` `--gradient_checkpointing` @Night1099 @Trav1slaflame @universewill
> Cc: [@PromeAIpro](https://github.com/PromeAIpro) using deepseek were experimental tries we had tested when writing this script, it was not included in code because it is not guaranteed work on any gpu...