Tom Dörr

Hi, thank you for your reply. I tried this but still have the same issue. I installed the newest sglang version directly from the repository ``` -e git+https://github.com/sgl-project/sglang.git@95c4e0dfac5a5f4a2f7f9292402fec26d0838f31#egg=sglang&subdirectory=python ``` prefill_token_logprobs:...

@m0g1cian Yes, I did that. The model generates the boolean value even if not constraint

I tried to extract the right logprob values from the `prefill_token_logprobs`. It seems to me like those are the logprobs of the generated tokens since the number of available token...

The issue seems to be that when I add an image to the prompt there are to many logprobs in prompt_logprobs. Since only the last logprob differs the resulting normalized...

Interesting. By the way the current version in the repo (main) has some changes to what logprob information you get. It also fails in the same way for me but...

Solved it by setting `mem-fraction-static` to `0.9`. Full command that works for me: ``` python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-34b --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 30813 --tp 2 --mem-fraction-static '0.9' ``` Maybe a...

Did you get an error message? 'service isn't ready!' usually just means that it's not yet done loading the model weights which can take a long time. For me loading...

"prixy=\n" or "proxy=\n"?

@arnavsinghvi11 Yes I know what you mean, I planned to add support to DSPy components after support for the clients is merged. Since it's a bigger refactor I would have...

#1002 should be solved by https://github.com/stanfordnlp/dspy/pull/1043 , but yes, they do look related