cuda10 2080Ti mutil_gpu solutions

[zyxcambridge]

1. horovod HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod

2. Open MPI
   3.0.4 no 2.1.1 wget http://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.1.tar.gz tar xzvf openmpi-2.1.1.tar.gz WORKDIR openmpi-2.1.1 ./configure make -j48 sudo make install sudo ldconfig which mpicc mpicc -show which mpiexec mpiexec --version mkdir OpenMpi wget http://www.open-mpi.org/papers/workshop-2006/hello.c mpicc hello.c -o hello mpirun --allow-run-as-root -np 4 ./hello 3.eno1 options="-np ${nb_gpus} -H localhost:${nb_gpus} -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x NCCL_SOCKET_IFNAME=eno1 -x NCCL_IB_DISABLE=1 -x LD_LIBRARY_PATH --mca btl_tcp_if_include eth1

4.gcc 4.9 g++ 4.9 ;others not ok