CrossPoint-DDP

CrossPoint-DDP copied to clipboard

Hello. I am currently using your code for an internship I am working on and noticed something about it. In the training loop, you seem to use the NTXentLoss function...

NadirBendo

Training so slow~~~

1

Hi，Thank you very much for your efforts, but when I ran it with three graphics cards, it took a very long time to run, and the GPU utilization did not...

YangParky

RuntimeError: [2] is setting up NCCL communicator and retreiving ncclUniqueId from [0] via c10d key-value store by key '1', but store->get('1') got error: Socket Timeout

4

Hello, Jerry Sun. Thank you for the sharing of your good implementation of DDP training for CrossPoint. When I was conducting the training, I met the issue: work = default_pg.allgather([tensor_list],...

dempsey-wen

RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=8, timeout=0:30:00)

5

Excuse me, when I was conducting distributed training, the log kept outputting "DEBUG SenderThread: 1236909 [sender. py: send(): 182] send: stats", and finally reported an RuntimeError: Timed out initializing process...

Lzyin