TensorRT-LLM
TensorRT-LLM copied to clipboard
NVIDIA
Provide an interface similar to OpenAI API
The most basic thing is just a simple chat/completions function similar to OpenAI.
The purpose is to facilitate access to existing applications that use the chatgpt API.
Currently, many open-source LLM projects have implemented this feature, such as the famous oobabooga/text generation webui
Users want something like this https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md, so they can switch their apps from OpenAI models to TRT-LLM models easily without code change.
Right now Python API still have a lot of issues to be fixed, I encapsulated one OpenAI API, but met #283, so you still need use C++ runtime, which means you need Triton. Spent weeks on TRT-LLM, it difficult to develop on python runtime.
@juney-nvidia What's the position of TRT-LLM's Python runime? I mean, python is easier than C++, and batch manager doesn't open source right now. Most developors may not use Triton since they won't meet that large commercial demand.
Sorry for replying late due to being trapped by other things.
Users want something like this https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md, so they can switch their apps from OpenAI models to TRT-LLM models easily without code change.
@Pevernow @merrymercy well received, will discuss with prod about this. @ncomly-nvidia for vis.
What's the position of TRT-LLM's Python runime? I mean, python is easier than C++, and batch manager doesn't open source right now. Most developors may not use Triton since they won't meet that large commercial demand.
@gesanqiu we have already released the Python binding of C++ runtime, including batch manager, does this fulfill your requirement here?
More info about openai chat completion API spec here https://github.com/openai/openai-openapi/tree/master
+1 for OpenAI API support, it's been 9 months since the pr requested :(
