Shi Shuai comments

Results 44 comments of


                                            Shi Shuai

[Question] How Sglang balances workload?

Pleace check https://docs.sglang.ai/router/router.html

In distributed scenarios, how to configure k8s ports and which ports need to be opened?

cc @ByronHsu

In distributed scenarios, how to configure k8s ports and which ports need to be opened?

Are you following https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker?

[Feature] support /v1/completions suffix parameter for completion

https://platform.openai.com/docs/api-reference/models/delete#:~:text=only%20supported%20for-,gpt%2D3.5%2Dturbo%2Dinstruct,-. I saw in the OpenAI API documentation that it seems to only support gpt-3.5-turbo-instruct. Is this a parameter that is about to be deprecated?

[Feature] support /v1/completions suffix parameter for completion

Thanks for providing this. It's good to have one. Why openai api only have this for gpt-3.5-turbo-instruct tho?

[Feature] support /v1/completions suffix parameter for completion

Thanks! That's helpful.

Error: Python.h

Can you try to use docker?

[Feature] Automatically truncate when the maximum tokens are exceeded instead of throwing an error

Thanks, we'll have a look of this

[Feature] Reorganize all the docs

1. Move quick_start.ipynb to Start folder, import the patch.py under the backend folder 2. remove redundant cmd 3. cover the print_highlight from the util.py in patch.py 4. remove block in...

Support for reasoning_content in API

I will review this @zhaochenyang20