[Bug] Can't use profiler in DP+EP

Open nannaer opened this issue 7 months ago • 0 comments

Checklist

[x] 1. I have searched related issues but cannot get the expected help.
[x] 2. The bug has not been fixed in the latest version.
[x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
[x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
[x] 5. Please use English, otherwise it will be closed.

Describe the bug

When use 16DP+16EP，can't use profiler in sglang

`[2025-05-12 21:31:17 DP12 TP12] Scheduler hit an exception: Traceback (most recent call last): File "/nvme1/chenjiefei/project/sglang/python/sglang/srt/managers/scheduler.py", line 2244, in run_scheduler_process scheduler.event_loop_overlap() File "/nvme1/chenjiefei/miniconda3/envs/sglang/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/nvme1/chenjiefei/project/sglang/python/sglang/srt/managers/scheduler.py", line 660, in event_loop_overlap self.process_input_requests(recv_reqs) File "/nvme1/chenjiefei/project/sglang/python/sglang/srt/managers/scheduler.py", line 886, in process_input_requests output = self._request_dispatcher(recv_req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nvme1/chenjiefei/project/sglang/python/sglang/utils.py", line 471, in call return fn(obj) ^^^^^^^ File "/nvme1/chenjiefei/project/sglang/python/sglang/srt/managers/scheduler.py", line 2031, in profile return self.stop_profile() ^^^^^^^^^^^^^^^^^^^ File "/nvme1/chenjiefei/project/sglang/python/sglang/srt/managers/scheduler.py", line 2097, in stop_profile self.torch_profiler.stop() File "/nvme1/chenjiefei/miniconda3/envs/sglang/lib/python3.12/site-packages/torch/profiler/profiler.py", line 809, in stop self._transit_action(self.current_action, None) File "/nvme1/chenjiefei/miniconda3/envs/sglang/lib/python3.12/site-packages/torch/profiler/profiler.py", line 852, in _transit_action action() File "/nvme1/chenjiefei/miniconda3/envs/sglang/lib/python3.12/site-packages/torch/profiler/profiler.py", line 239, in stop_trace self.profiler.exit(None, None, None) File "/nvme1/chenjiefei/miniconda3/envs/sglang/lib/python3.12/site-packages/torch/autograd/profiler.py", line 378, in exit self.kineto_results = _disable_profiler() ^^^^^^^^^^^^^^^^^^^ RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty.

[2025-05-12 21:31:17] Child process unexpectedly failed with an exit code 131. pid=2215951`

Reproduction

start a 16DP+16EP instance, and use curl http://localhost:30000/start_profile -H "Content-Type: application/json" to start profiler, use curl http://localhost:30000/stop_profile -H "Content-Type: application/json" to stop prodiler

Environment

Python: 3.12.0 CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA L20Y GPU 0,1,2,3,4,5,6,7 CUDA_HOME: /usr/local/cuda NVCC: Cuda CUDA Driver Version: 550.127.08 PyTorch: 2.6.0+cu124 sglang: 0.4.6.post2 sgl_kernel: 0.1.1 flashinfer_python: 0.2.5 triton: 3.2.0 transformers: 4.51.1 torchao: 0.10.0 numpy: 2.2.5 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.30.2 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.18 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.4 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.2 uvloop: 0.21.0 vllm: Module Not Found xgrammar: 0.1.19 openai: 1.75.0 tiktoken: 0.9.0 anthropic: 0.50.0 litellm: 1.68.0 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU0 X GPU1 NV8 GPU2 NV8 NV8 GPU3 NV8 NV8 NV8 GPU4 NV8 GPU5 NV8 GPU6 NV8 GPU7 NV8 NIC0 PIX NIC1 NODE PIX NIC2 NODE NODE PIX NIC3 NODE NIC4 SYS NIC5 SYS NIC6 SYS NIC7 SYS | packaged by Anaconda, Inc. | (main, Oct 2 2023, 17:29:18) [GCC 11.2.0] Compute Capability: 9.0 compilation tools, release 12.4, V12.4.99 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID NV8 NV8 NV8 NV8 NV8 NV8 NV8 PIX NODE NODE NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A X NV8 NV8 NV8 NV8 NV8 NV8 NODE PIX NODE NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A X NV8 NV8 NV8 NV8 NV8 NODE NODE PIX NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A X NV8 NV8 NV8 NV8 NODE NODE NODE PIX SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS PIX NODE NODE NODE 1,3,5,7,9,11 1 N/A NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS NODE PIX NODE NODE 1,3,5,7,9,11 1 N/A NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS NODE NODE PIX NODE 1,3,5,7,9,11 1 N/A NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS NODE NODE NODE PIX 1,3,5,7,9,11 1 N/A NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7

ulimit soft: 65535

May 12 '25 13:05 nannaer