Bluefog-Lib
Proposal for local GPU communication merging
Current win_ops logics is
win_create -> gradient/iterate update -> win_put -> win_sync
The processing between all nodes/agents are almost decoupled and independent.
We want to further optimize our communication for multi machines cases. We know the communication between multiple GPUs within in same physical machine should be faster than communication between different machines. Further, we can utilize the NCCL, RDMA, etc technique to accelerate the speed. I suggest modifying the processes into
Local machine leader:
win_create -> gradient/iterate update -> Local Allreduce -> win_put -> win_sync
ocal machine worker1:
nothing ----> gradient/iterate update -> Local Allreduce ---- nothing
local machine woker2:
nothing ----> gradient/iterate update -> Local Allreduce ---- nothing
