vector-quantize-pytorch icon indicating copy to clipboard operation
vector-quantize-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

How to gather cluster_size and embed_sum when use DataParallel (DP)

Open liuyoude opened this issue 3 years ago • 1 comments

I notice the code can gather data for EMA in DDP, but I filed it when I use DP, because distributed.all_reduce need distributed.init_process_group firstly. How to gather data in DP

liuyoude avatar Jan 28 '22 10:01 liuyoude

In DP data cannot be gathered. Please run the same job you run in DP with DDP. I had the same issues in my work, there is no workaround.

danieltudosiu avatar May 02 '22 18:05 danieltudosiu