YunTianming2024 comments

Repositories
Issues
Comments

Results 1 comments of


                                            YunTianming2024

[BUG] Issue with using optimise_hyperparameter with PyTorch DDP

try to add "NCCL_P2P_DISABLE=1" in your cmd. I tried this to my cmd and "CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NCCL_P2P_DISABLE=1 torchrun --nnode=1 --nproc_per_node=8 --master_port=12345 run.py" can work