Error launching model on Triton on multigpu nodes

[sujituk]

Background: Setup GKE node pool with 2 H100 nodes (8GPUs each) and required NFS storage. Trying to serve Llama3 405B model after checkpoint conversion and building TRT-LLM engine.

Environment: Triton image: nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3 Triton version: 0.16 mpirun (Open MPI) 4.1.5rc2

Issue: Launching on leader node using command line: python3 /var/run/models/tensorrtllm_backend/scripts/launch_triton_server.py --model_repo=/var/run/models/tensorrtllm_backend/triton_model_repo --world_size 16

it fails with error:

---

Triton model repository is at:'/var/run/models/tensorrtllm_backend/triton_model_repo' Server is assuming each node has 8 GPUs. To change this, use --gpu_per_node Executing Leader (world size: 16) Begin waiting for worker pods.

kubectl get pods -n default -l leaderworkerset.sigs.k8s.io/group-key=<redacted> --field-selector status.phase=Running -o jsonpath='{.items[*].metadata.name}' 'triton-trtllm-0 triton-trtllm-0-1' 2 of 2.0 workers ready.

ORTE was unable to reliably start one or more daemons. This usually is caused by:

• not finding the required libraries and/or binaries on one or more nodes. Please check your PATH and LD_LIBRARY_PATH settings, or configure OMPI with --enable-orterun-prefix-by-default

• lack of authority to execute on one or more specified nodes. Please verify your allocation and authorities.

• the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base). Please check with your sys admin to determine the correct location to use.

• compilation of the orted with dynamic libraries when static are required (e.g., on Cray). Please check your configure cmd line and consider using one of the contrib/platform definitions for your system type.

• an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements).

---

[triton-trtllm-0:00238] Job UNKNOWN has launched [triton-trtllm-0:00238] [[41321,0],0] Releasing job data for [41321,1] [triton-trtllm-0:00238] sess_dir_finalize: proc session dir does not exist [triton-trtllm-0:00238] sess_dir_finalize: job session dir does not exist [triton-trtllm-0:00238] sess_dir_finalize: jobfam session dir does not exist [triton-trtllm-0:00238] sess_dir_finalize: jobfam session dir does not exist [triton-trtllm-0:00238] sess_dir_finalize: top session dir does not exist [triton-trtllm-0:00238] sess_dir_cleanup: job session dir does not exist [triton-trtllm-0:00238] sess_dir_cleanup: top session dir does not exist [triton-trtllm-0:00238] [[41321,0],0] Releasing job data for [41321,0] [triton-trtllm-0:00238] sess_dir_cleanup: job session dir does not exist [triton-trtllm-0:00238] sess_dir_cleanup: top session dir does not exist exiting with status 1 Waiting 15 second before exiting.

Launched on non-leader node Triton model repository is at:'/var/run/models/tensorrtllm_backend/triton_model_repo' Worker paused awaiting SIGINT or SIGTERM. Verified: mpirun is on the path.

Question: mpirun works fine in a single node. Is there any configuration that needs to be done when mpirun spans multiple nodes.