Yixin Dong
Yixin Dong
Thanks for the report @dtkettler Currently there are several issues with llama3 because it changes the tokenizer a lot. That will be fixed soon in these days
Hi @digisomni, thanks for reporting the error! Could you provide the complete error message and the script to reproduce the error so we can better identify the problem? I failed...
@GunjanDhanuka The tokenizer issue is solved in this PR: #2649. Please tell me if there are any other related problems!
That is related to a recent change in tokenizer in #2416. We will fix that soon
> Thanks, I test your impl use Llama3.1-8b-Instruct and [eagle model](https://huggingface.co/yuhuili/EAGLE-LLaMA3.1-Instruct-8B). When set `export SGLANG_ENABLE_SPEC_V2=0`, the response satisfies `r"^user@example\.com$"` . When set `export SGLANG_ENABLE_SPEC_V2=1`, the response is `use the following...
> @Ubospica Can help review the impl? Certainly. I will check that out. Thanks for the reference!
Now all the tests have passed. cc @hnyls2002
Hi @zlwu92, sorry for the late response. XGrammar has already be integrated into sglang and vllm, so it should be easy to benchmark with and without structured generation with their...
@zlwu92 I think so. You can test it with sglang's benchmark script.