polestar

When using gradient accumulation, does the order of optimizer.zero_grad() affect training?

2

if I use accelerate+deepspeed to train a model, and I set deepspeed_config: gradient_accumulation_steps: 8 offload_optimizer_device: cpu offload_param_device: cpu zero3_init_flag: false zero_stage: 2 does the order of the order of backward(),...