ckhfor

未来に向かう。

Results 2 comments of


                                            ckhfor

Now I use lm-format-forcer+vllm and it takes 10 seconds to generate a json. This speed is not feasible in my current business. Is there any accelerated solution, such as a paper, plan or pr?

python -> cpp, python is too slow... multithreading, python gil... pre-compute accept/reject/uncertain ... many ways to accelerate

Add utf-8 character support to JsonSchemaParser KEYS

Can we change the character based parser to token_id based?