vLLM support?

[lhl]

The docs mention that you used vLLM for inferencing, but it looks like Orion support hasn't been upstreamed yet: https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models

Can you share the model file or do you have an ETA for upstreaming the code? HF transformers inferencing is slow enough to make Orion pretty unusable even for running evals.

[ZeroYuJie]

I've been using the Orion branch from https://github.com/dachengai/vllm and it's running, but there might be issues with outputs in different languages

[shuiqingliu]

I've been using the Orion branch from https://github.com/dachengai/vllm and it's running, but there might be issues with outputs in different languages

Yeap ,I am trying to translate from Chinese to English, but the output still contains Chinese characters. 😭