Bing Han

When will v2 version be completed? Any new updates?

Similar issue here. ```python In [4]: langcodes.get("ko").language_name() Out[4]: 'Korean' In [5]: langcodes.get("kor_Hang").language_name() Out[5]: 'Korean' In [6]: langcodes.closest_match("ko", ["kor_Hang"]) Out[6]: ('und', 1000) ```

不知道官方是否能适配到如 - [text-generation-inference](https://github.com/huggingface/text-generation-inference) - [vllm](https://github.com/vllm-project/vllm) 这样的推理引擎项目上去呢。

你好，请问你解决这个问题了么，会不会是后面一部分是预留的token位置？

Deploying qwen to TGI encounters this error ``` 2024-01-05T11:20:53.653089Z ERROR text_generation_launcher: Error when initializing model Traceback (most recent call last): File "/opt/conda/bin/text-generation-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311,...

目前api文档是widdershins自动生成的，可能有些问题，后序会进行更新。可以启动api服务后先使用 http://127.0.0.1:7861/docs访问fastapi网页文档

Hi, @gwenzek, could you please help me review this PR?

+1 here.

But iterm2 doesn't have this problem, how is it done?

Hello, I have a similar question. In their reported results  In their results, the tokenized bleu is reported, and the tokenizer used in different articles should be different, so...