sglang icon indicating copy to clipboard operation
sglang copied to clipboard

Published 20 hours ago • verified publisher icon sgl-project

[Bug]

Open zxsimple opened this issue 1 week ago • 1 comments

Checklist

  • [x] 1. I have searched related issues but cannot get the expected help.
  • [x] 2. The bug has not been fixed in the latest version.
  • [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • [x] 5. Please use English, otherwise it will be closed.

Describe the bug

Serving DeepSeek-R1 model on 2 * 8 H20 nodes crashes after a period of normal serving. This happened twice, once for minutes, another for 9 hours.

[2025-02-18 23:24:03 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 107, cache hit rate: 49.62%, token usage: 0.79, #running-req: 99, #queue-req: 0
[2025-02-18 23:24:04 TP0] Decode batch. #running-req: 100, #token: 399970, token usage: 0.80, gen throughput (token/s): 753.26, #queue-req: 0
[2025-02-18 23:24:06] INFO:     29.119.83.212:60786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-02-18 23:24:06 TP0] Prefill batch. #new-seq: 1, #new-token: 163, #cached-token: 29, cache hit rate: 49.60%, token usage: 0.78, #running-req: 99, #queue-req: 0
[2025-02-18 23:24:09 TP0] Decode batch. #running-req: 100, #token: 396663, token usage: 0.79, gen throughput (token/s): 779.65, #queue-req: 0
[2025-02-18 23:24:12] INFO:     29.119.83.212:60760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-02-18 23:24:12 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 191, cache hit rate: 49.63%, token usage: 0.79, #running-req: 99, #queue-req: 0
Fatal Python error: Segmentation fault

Thread 0x00007fd293fff700 (most recent call first):
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 462 in watchdog_thread
  File "/usr/lib64/python3.11/threading.py", line 982 in run
  File "/usr/lib64/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007fd2937fe700 (most recent call first):
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 527 in forward
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 773 in forward
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 835 in forward
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 874 in forward
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 761 in forward_extend
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 796 in forward
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 164 in forward_batch_generation
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 140 in forward_thread_func_
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 109 in forward_thread_func
  File "/usr/lib64/python3.11/threading.py", line 982 in run
  File "/usr/lib64/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007fe5d39d0700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 331 in wait
  File "/usr/lib64/python3.11/threading.py", line 629 in wait
  File "/usr/local/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib64/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007fe5d31cf700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 331 in wait
  File "/usr/lib64/python3.11/threading.py", line 629 in wait
  File "/usr/local/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib64/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007fe7acd72700 (most recent call first):
  File "/usr/local/lib64/python3.11/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 47 in _recv_msg
  File "/usr/local/lib64/python3.11/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 153 in _read_thread
  File "/usr/lib64/python3.11/threading.py", line 982 in run
  File "/usr/lib64/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007feb5bbb5740 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 327 in wait
  File "/usr/lib64/python3.11/threading.py", line 629 in wait
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 171 in resolve_batch_result
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1257 in process_batch_result_decode
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1116 in process_batch_result
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 519 in event_loop_overlap
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825 in run_scheduler_process
  File "/usr/lib64/python3.11/multiprocessing/process.py", line 108 in run
  File "/usr/lib64/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/lib64/python3.11/multiprocessing/spawn.py", line 135 in _main
  File "/usr/lib64/python3.11/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, uvloop.loop, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, psutil._psutil_posix, setproctitle, zmq.backend.cython._zmq, yaml._yaml, markupsafe._speedups, PIL._imaging, PIL._imagingft, msgspec._core, sentencepiece._sentencepiece, regex._regex, msgpack._cmsgpack, google._upb._message, ray._raylet, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.typing.builtins.itertools, numba.cpython.builtins.math, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, xxhash._xxhash, pyarrow._json, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, cuda_utils, __triton_launcher (total: 119)
TENCENT64:39455:43935 [3] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39455:41862 [3] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39455:41737 [3] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39459:43927 [7] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39459:41856 [7] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39459:41727 [7] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39458:43932 [6] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39458:41857 [6] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39458:41729 [6] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39454:43931 [2] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39454:41865 [2] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39454:41735 [2] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39456:43930 [4] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39456:41859 [4] NCCL INFO [Service thread] Connection closed by localRank 0
TENCENT64:39456:41728 [4] NCCL INFO [Service thread] Connection closed by localRank 0
[2025-02-18 23:24:14 TP8] Scheduler hit an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 493, in event_loop_overlap
    recv_reqs = self.recv_requests()
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 575, in recv_requests
    recv_reqs = broadcast_pyobj(recv_reqs, self.tp_rank, self.tp_cpu_group)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/utils.py", line 693, in broadcast_pyobj
    dist.broadcast(tensor_size, src=src, group=dist_group)
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2425, in broadcast
    work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [29.119.83.212]:11145

[2025-02-18 23:24:14 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 493, in event_loop_overlap
    recv_reqs = self.recv_requests()
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 575, in recv_requests
    recv_reqs = broadcast_pyobj(recv_reqs, self.tp_rank, self.tp_cpu_group)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/utils.py", line 693, in broadcast_pyobj
    dist.broadcast(tensor_size, src=src, group=dist_group)
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2425, in broadcast
    work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [29.119.83.212]:3761

[2025-02-18 23:24:14 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 493, in event_loop_overlap
    recv_reqs = self.recv_requests()
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 575, in recv_requests
    recv_reqs = broadcast_pyobj(recv_reqs, self.tp_rank, self.tp_cpu_group)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/utils.py", line 693, in broadcast_pyobj
    dist.broadcast(tensor_size, src=src, group=dist_group)
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2425, in broadcast
    work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [29.119.83.212]:11145

[2025-02-18 23:24:14] Received sigquit from a child proces. It usually means the child failed.
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib64/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 343, in sigquit_handler
    logger.error(
  File "/usr/lib64/python3.11/logging/__init__.py", line 1518, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib64/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/usr/lib64/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/usr/lib64/python3.11/logging/__init__.py", line 1118, in emit
    self.handleError(record)
  File "/usr/lib64/python3.11/logging/__init__.py", line 1031, in handleError
    sys.stderr.write('--- Logging error ---\n')
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Call stack:
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/entrypoints/http_server.py", line 524, in launch_server
    uvicorn.run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 579, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 66, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib64/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 343, in sigquit_handler
    logger.error(
Message: 'Received sigquit from a child proces. It usually means the child failed.'

Reproduction

export NCCL_NET=IB export NCCL_SOCKET_IFNAME=bond1 export GLOO_SOCKET_IFNAME=bond1 export NCCL_DEBUG=INFO

node 1 python3.11 -m sglang.launch_server --model-path /apdcephfs_gy4/share_303286581/hunyuan/xishengzhao/deepseek-r1/DeepSeek-R1
--tp 16
--dist-init-addr ${MASTER_NODE}:5000
--nnodes 2
--node-rank 0
--trust-remote-code
--host 0.0.0.0
--port 30000

node 2 python3 -m sglang.launch_server --model-path /apdcephfs_gy4/share_303286581/hunyuan/xishengzhao/deepseek-r1/DeepSeek-R1
--tp 16
--dist-init-addr ${MASTER_NODE}:5000
--nnodes 2
--node-rank 1
--trust-remote-code
--host 0.0.0.0
--port 30000

Environment

Python: 3.11.9 (main, Sep 24 2024, 12:12:02) [GCC 8.5.0 20210514 (Tencent 8.5.0-23)] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H20 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 535.161.08 PyTorch: 2.5.1+cu124 sglang: 0.4.3.post2 sgl_kernel: 0.0.3.post6 flashinfer: 0.2.1.post2+cu124torch2.5 triton: 3.1.0 transformers: 4.48.3 torchao: 0.8.0 numpy: 1.26.4 aiohttp: 3.11.12 fastapi: 0.115.8 hf_transfer: 0.1.9 huggingface_hub: 0.28.1 interegular: 0.3.3 modelscope: 1.23.0 orjson: 3.10.15 packaging: 24.1 psutil: 7.0.0 pydantic: 2.9.2 multipart: 0.0.20 zmq: 26.2.1 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.6.4.post1 openai: 1.63.2 tiktoken: 0.9.0 anthropic: 0.45.2 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16 NIC17NIC18 NIC19 NIC20 NIC21 NIC22 NIC23 NIC24 NIC25 NIC26 NIC27 NIC28 NIC29 NIC30 NIC31 NIC32 NIC33 NIC34 NIC35 NIC36 NIC37 NIC38 NIC39 NIC40 NIC41 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PHB PIX SYS SYS SYS SYS 0-95,192-287 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX PHB SYS SYS SYS SYS 0-95,192-287 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE PIX NODE 96-191,288-383 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE PIX NODE NODE 96-191,288-383 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PHB NODE NODE PIX 96-191,288-383 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PIX NODE NODE PHB 96-191,288-383 1 N/A NIC0 SYS SYS SYS SYS NODE NODE NODE NODE X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC1 SYS SYS SYS SYS NODE NODE NODE NODE PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC2 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC3 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC4 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC5 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC6 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC7 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC8 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC9 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC10 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC11 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC12 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC13 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC14 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC15 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC16 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC17 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC18 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC19 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC20 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC21 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC22 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC23 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC24 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC25 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC26 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC27 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC28 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC29 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC30 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC31 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC32 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYS SYS NODE NODE NODE NODE NIC33 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYS SYS NODE NODE NODE NODE NIC34 PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC35 NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC36 NODE PHB PIX NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X PHB SYS SYS SYS SYS NIC37 NODE PIX PHB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PHB X SYS SYS SYS SYS NIC38 SYS SYS SYS SYS NODE NODE PHB PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS X NODE NODE PHB NIC39 SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC40 SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE NIC41 SYS SYS SYS SYS NODE NODE PIX PHB NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PHB NODE NODE X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: NIC41 CPU Affinity NUMA Affinity GPU NUMA ID NIC1: MA ID NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 NIC10: mlx5_10 NIC11: mlx5_11 NIC12: mlx5_12 NIC13: mlx5_13 NIC14: mlx5_14 NIC15: mlx5_15 NIC16: mlx5_16 NIC17: mlx5_17 NIC18: mlx5_18 NIC19: mlx5_19 NIC20: mlx5_20 NIC21: mlx5_21 NIC22: mlx5_22 NIC23: mlx5_23 NIC24: mlx5_24 NIC25: mlx5_25 NIC26: mlx5_26 NIC27: mlx5_27 NIC28: mlx5_28 NIC29: mlx5_29 NIC30: mlx5_30 NIC31: mlx5_31 NIC32: mlx5_32 NIC33: mlx5_33 NIC34: mlx5_bond_1 NIC35: mlx5_bond_2 NIC36: mlx5_bond_3 NIC37: mlx5_bond_4 NIC38: mlx5_bond_5 NIC39: mlx5_bond_6 NIC40: mlx5_bond_7 NIC41: mlx5_bond_8

ulimit soft: 1000000

zxsimple avatar Feb 19 '25 02:02 zxsimple