server Support for vLLM and TRT-LLM running in OpenAI compatible mode

Is your feature request related to a problem? Please describe. I'd like to be able to run vLLM emulating the OpenAI compatible API to use vLLM as a drop-in replacement of ChatGPT.

Describe the solution you'd like I'd like Triton allow me run vLLM as indicated in vLLM documentation

Example:

python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

Describe alternatives you've considered It is possible to use the REST API, however, for developers already leveraging OpenAI and serving open-source LLMs using the OpenAI API would allow a faster path for replacement of OpenAI

Nov 15 '23 19:11 vecorro

Thanks for submitting feature request! CC @nnshah1 on the request for starting server with the OpenAI compatible API.

For the client side, we have introduced the generate endpoint. which is an OpenAI-like endpoint support and can improve Triton adoption for LLM use cases.

Nov 16 '23 19:11 krishung5

@vecorro It's a bit challenging as the OpenAI API is a moving target and already the completion api is legacy and out of date with many applications. As triton as an inference server serves many types of inference we've focused on providing ways to customize endpoints also with the expectation that many actual deployments would use an additional api gateway to translate from a service api to the inference api (for example langchain).

that being said - your perspective is important. Is the legacy completion api enough or would we need to support the newer apis as well? Do you think integration with a project like langchain or other llm projects would be a viable approach or would triton need to provide the interface directly?

Nov 16 '23 20:11 nnshah1

@nnshah1

that being said - your perspective is important. Is the legacy completion api enough or would we need to support the newer apis as well? Do you think integration with a project like langchain or other llm projects would be a viable approach or would triton need to provide the interface directly?

The key here, to be usefull to the community, is actuality & compatibility. every other often used tool, like LiteLLM (middlware/API-Proxy) or loaders, like ooba's TGI, vLLM, FastChat are on the run to implement functions and to pick up with v1, since this is essential for local LLMs to be a replacement for the costly openai-stuff. There are a few very popular projects which use the very latest features heavly, like Mircosofts Autogen and MemGPT.

IMHO: I have been testing all these and just had a peek into triton, but already assume, it would be the much supirior solution of running multiple LLMs locally - IF there would be an out of the box compatibility with the openAI-API. Maybe teaming up with https://litellm.ai/ is a quick solution.

Nov 18 '23 09:11 chymian

@chymian @nnshah1 im the litellm maintainer - what do you need from us ?

Nov 20 '23 23:11 ishaan-jaff

I agree with @nnshah1; the key is actuality and compatibility, and I would add ease of use. For experimentation/ded purposes, vLLM is excellent as it allows you to try an LLM without the complexity and time required by TensorRT-LLM. Making the existing vLLM implementation of the legacy OpenAI API can be good enough; meanwhile, vLLM evolves to support newer versions of the OpenAI API. I'm on the VMware team working with NVIDIA on the private AI initiative. From conversations with customers looking for ways to run LLMs on-premises, they are getting started on that journey, so a quick drop-in replacement of OpenAI's models would facilitate things for them. Thanks all!

Nov 21 '23 13:11 vecorro

@npuichigo has mentioned an integration that could be useful here:

https://github.com/NVIDIA/TensorRT-LLM/discussions/591

Dec 06 '23 14:12 nnshah1

@npuichigo has mentioned an integration that could be useful here:

NVIDIA/TensorRT-LLM#591

https://github.com/npuichigo/openai_trtllm provides an OpenAI-like API for trtllm triton backend, but I think vllm in triton would be something alike.

Dec 07 '23 08:12 npuichigo

Closing issue due to inactivity. Please reopen if you would like to follow up with this issue.

Feb 20 '24 17:02 dyastremsky

Hello,

Seems it's supporting vllm now: https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton

But can we use the vllm OpenAI APIs ?

Thanks a lot

Mar 22 '24 11:03 BodhiHu

We don't currently support it directly - but are still thinking through the best ways to add a compatible API in a way that we can maintain.

Mar 22 '24 19:03 nnshah1

Thank you for the clearance~~ though I think we could simply adapte the vLLM vllm.entrypoints.openai.api_server to the triton http endpoint.

Mar 23 '24 02:03 BodhiHu

is there any workaround so far ?

Thank you for the clearance~~ though I think we could simply adapte the vLLM vllm.entrypoints.openai.api_server to the triton http endpoint.

Apr 16 '24 08:04 panpan0000

@chymian @nnshah1 im the litellm maintainer - what do you need from us ?

@ishaan-jaff

Is there a guide on adding an llm provider to litellm or a proscribed starting point / skeleton?

Apr 27 '24 12:04 nnshah1

Is this planned for any specific release?

Aug 12 '24 14:08 anubhav-agrawal-mu-sigma

@anubhav-agrawal-mu-sigma - we are currently planning a tutorial showcasing how to create an open ai compatible triton server using triton's in-process python api - that tutorial is planned for September time frame.

Aug 12 '24 14:08 nnshah1

@nnshah1 : I am interested in the mentioned tutorial, please let me know when you have it online. Many thanks !