no_sync equivalent used for gradient accumulation

[amsword]

In gradient accumulation, we do not need to gather the gradient for the first N - 1 iterations. If it is pytorch/DDP, we can use the no_sync() as follows. In apex DDP, is there any equivalent?

https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync

[yushanyong]

@amsword Have you solved this problem？

[zhengyuan-xie]

@amsword @601222543 I have the same problem, do you have any solution? Thanks.