YanjieLiang

Training error with multiple nodes using LaSOT

2

Training errors with multiple nodes using LaSOT: [E ProcessGroupNCCL.cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1800265 milliseconds before timing out. [E ProcessGroupNCCL.cpp:341] Some NCCL operations...