[Feature Request] Add support for LiteLLM

[tan-yong-sheng]

Is your feature request related to a problem? Please describe

Hi admin

I have commented here: https://github.com/Filimoa/open-parse/issues/10#issuecomment-2214343987, but after that, I found here could be a better place to put this comment:

I would like to suggest adding support for Litellm.

Litellm is an open source project, which unifies the API call of 100+ LLMs (including Anthropic, Cohere, and Ollama, etc) in an OpenAI compatible format: https://github.com/BerriAI/litellm

I believe integrating Litellm would be a fantastic enhancement because people could choose to switch or use their preferred embedding model api instead of OpenAI's ones only when dealing with semantic processing. Thanks.

Describe the solution you'd like

For example, if they used litellm python client and without self hosting litellm proxy, then their code could be like this (which is very consistent with OpenAI python client format):

Reference: https://github.com/BerriAI/litellm

if someone self hosted litellm proxy, which they can call LLM API in an OpenAI compatible format via llmlite proxy, you could see the code could be as follows:

Reference: https://litellm.vercel.app/docs/providers/azure_ai#passing-additional-params---max_tokens-temperature

You can see if someone self host the litellm proxy, he will only need to change the openai's base url, and all the other code are similar as openai's one...

Reference: https://litellm.vercel.app/docs/proxy/user_keys
import openai
client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
    extra_body={ # pass in any provider-specific param, if not supported by openai, https://docs.litellm.ai/docs/completion/input#provider-specific-params
        "metadata": { # 👈 use for logging additional params (e.g. to langfuse)
            "generation_name": "ishaan-generation-openai-client",
            "generation_id": "openai-client-gen-id22",
            "trace_id": "openai-client-trace-id22",
            "trace_user_id": "openai-client-user-id2"
        }
    }
)

print(response)

There are also quite a few projects that used litellm: https://litellm.vercel.app/docs/project to call models from different providers on LiteLLM.

Hope for consideration, thanks.

Related component

Libraries

Describe alternatives you've considered

No response

Additional context

No response

[dblock]

[Catch All Triage - 1, 2, 3]