Jinn comments

Results 27 comments of


                                            Jinn

JSON Mode

I am meeting with the same issue, any advice on how to solve it?

How to solve GPT-4 turbo (gpt-4-1106-preview) error

Thank you for the advice, but I got a question that since the file DataType.cs is part of the Unity Package Cache. And any changes made will be lost. Does...

[Bug] Eagle mtbench benchmark error

Thank you for mentioning this. cc @zhyncs . Could you help take a look on this EAGLE issue? Thx a lot!

[Bug] sglang-router curl get return without `content-type: application/json` in the header

Thanks for checking, we've already let the router team know.

Why is the throughput of A40 15 token/s, slower than vllm?

According to the docker logs, it seems to me that you need to update the GPU driver to match the CUDA toolkit. And also you can align PyTorch/CUDA versions and...

[Feature Request] Accepting multiple weight updates in update_weights_from_distributed

Cool. You are pretty welcome to work on it! Please lmk if you meet with any problem or when u have a complete PR ready to go.

[Bug] Vision attention mask cache is never released and cause OOM

Appreciate that for your question and proposal! We will let the VLM team know. cc @mickqian Could you take a look on that? Thx!

[Bug] update_weights_from_tensor raise EOFError when TP>1

> An EOFError error was raised when using update_weights_from_tensor at TP>4, it seens the data deserialize before the full data received. > `tp_size = 4` So are u trying to...

[Bug] update_weights_from_tensor raise EOFError when TP>1

That's awesome! Would you mind raising a PR about fixing this issue?

[Bug] sglang doesn't stop the generation when the request is canceled

Thank you for your suggestion and kind words! We will try to introduce a cancellation token or flag within the client-server communication. When the client cancels the request, this token...