Juwan Yoo

Not sure if this could be helpful or not, but I ran [llmperf](https://github.com/ray-project/llmperf) for both [main branch](https://github.com/sgl-project/sglang/commit/9208591f05c39963f423fb3fee841f94276da187) and [incoming branch](https://github.com/ispobock/sglang/commit/a4c12f49e295de8b841b3d9c3d0a8dd788c56df7). Overall this PR seems to make things much faster: -...

@81549361 Startup command I used for both are the same: ``` python3 -m sglang.launch_server \ --model-path "${MODEL}" \ --host 127.0.0.1 \ --port 8080 \ --context-length "4096" \ --max-prefill-tokens "16384" \...

Thanks for the fix. but I am seeing another error: ``` Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86,...