How to Enable Multi-GPU Inference on a Single Machine with EXO

[svm87601]

Hello! I am using EXO for distributed inference and would like to utilize multiple GPUs on a single machine to speed up inference. I have successfully configured EXO to work with a single GPU, but I'm not sure how to enable multi-GPU usage on the same machine. Could someone please guide me on how to set it up so that multiple GPUs are used for inference?

Environment Information:

• Operating System: Ubuntu 22.04
• CUDA Version: 12.5
• Driver Version: 555.42.02
• GPUs: RTX 3090 × 2

[hpvd]

possibly duplicate/related to How to use multiple GPU from a node https://github.com/exo-explore/exo/issues/411