Abulhair Saparov

Multi-node inference with Bloom: Unhandled CUDA error in ProcessGroupNCCL.cpp (called from all_reduce in torch)

3

I am trying to get multi-node inference working with 4 nodes, each with 4xRTX8000 GPUs (48GB per GPU). `deepspeed --hostfile=$hostfile Megatron-DeepSpeed/scripts/inference/bloom-ds-inference.py --name bigscience/bloom`. ([source for the script is here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/bloom-inference/scripts/inference/bloom-ds-inference.py)) The...

Multi-node inference with Bloom: Unhandled CUDA error in ProcessGroupNCCL.cpp (called from all_reduce in torch)

31

I am trying to get multi-node inference working with 4 nodes, each with 4xRTX8000 GPUs (48GB per GPU). `deepspeed --hostfile=$hostfile Megatron-DeepSpeed/scripts/inference/bloom-ds-inference.py --name bigscience/bloom`. The script finishes loading all the checkpoints...