BerriAI
[Feature]: Adding support for Volcano Engine
The Feature
I am using LiteLLM to proxy request for different providers.
Motivation, pitch
I am using Volcano Engine internally https://www.volcengine.com/docs/82379/1133189#python and also OpenAI compatible services, I do want to use liteLLM as the proxy layer to aggregate different providers. Now, one blocker is it miss the support for volcano engine.
Twitter / LinkedIn details
https://www.linkedin.com/in/jiaxin-shan/
I check the code and seems the prompt part needs to construct the prompt using chat template and special tokens. Is there a way to hornor the original chat template? If so, that would be easier and clean.
Hey @Jeffwan this is the curl i see from docs
it looks like volcano should already respect the messages format right (no prompt formatting needed)?
// Request
curl --request POST \
--url http://maas-api.ml-platform-cn-beijing.volces.com/api/v2/endpoint/${YOUR_ENDPOINT_ID}/chat \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
"stream": false,
"parameters": {
"max_new_tokens": 1024,
"temperature": 0.9
},
"messages": [
{
"role": "user",
"content": "你好"
},
{
"role": "assistant",
"content": "你好，有什么可以帮助您？"
},
{
"role": "user",
"content": "你可以做些什么？"
}
]
}'
// Response
{
"choices": [
{
"message": {
"role": "assistant",
"content": "我可以回答各种问题，例如历史、科学、技术、文化、娱乐等方面的问题。我还可以生成文本，例如摘要、文章、故事等。您需要我做什么呢？"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 43,
"total_tokens": 63
}
}
the only blocker to using it today w/ litellm is the fact that their api endpoint is not /chat/completions but instead just /chat?
Hey @Jeffwan is this still an ongoing blocker?
Happy to do a quick call on this to speed up the PR
https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
@krrishdholakia sorry for late. It is, it uses different SDKs, role definitions, I kind of think we need to add a new model wrapper. I scheduled a sync up with you.
