New Models/Endpoints/Providers

[krrishdholakia]

Parent issue to track new models/endpoints/providers to add to litellm, comment below for new ones

• [x] Vertex AI Mistral - https://github.com/BerriAI/litellm/issues/4874
• [x] Vertex AI Codestral support - https://github.com/BerriAI/litellm/issues/4984 s/o @allen-cook
• [x] Azure Batches API support - https://github.com/BerriAI/litellm/issues/5073
• [x] https://github.com/BerriAI/litellm/issues/5213 Add Gemini Context Caching in Anthropic Format
• [x] Gemini Context Caching - https://github.com/BerriAI/litellm/issues/4284
• [ ] Reka AI - https://github.com/BerriAI/litellm/issues/3607
• [x] Use Bedrock Guardrails https://github.com/BerriAI/litellm/issues/4938
• [x] Vertex AI Multimodal embeddings - https://github.com/BerriAI/litellm/issues/4622
• [ ] Fake streaming json mode for groq models - https://github.com/BerriAI/litellm/issues/4804
• [x] Amazon Multimodal embedding models - https://github.com/BerriAI/litellm/issues/4353
• [ ] Support Bedrock document understanding
• [ ] Support using s3 buckets for gemini vision calls (requires uploading files to gcs for larger files)
• [x] Prefill standardization - https://github.com/BerriAI/litellm/issues/4881
• [x] #4952
• [x] #4953
• [x] #4954
• [ ] #4955
• [x] https://github.com/BerriAI/litellm/issues/4966
• [x] Azure gpt-4o-mini support - s/o @seonghobae
• [x] #5017
• [x] Natively support pass-through endpoints for vertex ai
• [x] Github provider support - https://github.com/BerriAI/litellm/commit/4258295a079cc2b596190d106b76b53635c51059, s/o @lazyhope
• [x] #5053

[guleng]

@krrishdholakia I have deployed the text graph model locally, but I am unable to configure it for use in Litellm. Besides Azure and other vendors' locally deployed text graph models, cannot they be used in Litellm?

Can you help me with how to configure it?

[seonghobae]

Please checkout Azure GPT-4o-mini ;) https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/openai-s-gpt-4o-mini-now-available-in-api-with-vision/ba-p/4200640

[guleng]

@seonghobae Does it not support locally deployed Wenshengtu models? Does it only support models from Azure and OpenAI manufacturers?

[krrishdholakia]

@guleng - your error message from the provider shows a 404 not found error.

• Is your provider openai-compatible? if so use this - https://docs.litellm.ai/docs/providers/openai_compatible
• If custom api, use this - https://docs.litellm.ai/docs/providers/custom_llm_server

If error continues, file a separate issue for this - with config.yaml, steps to repro, and error traceback received.

[lazyhope]

Github just released a new inference endpoint: https://models.inference.ai.azure.com, it has free tier for popular models like gpt-4o so would be worth adding to litellm.

Marketplace: https://github.com/marketplace/models

[krrishdholakia]

added @lazyhope

[lazyhope]

added @lazyhope

Awesome! Thank you!

[yigitkonur]

Hey Llama 3.1 - 405B is added to AWS Bedrock - model ID is meta.llama3-1-405b-instruct-v1:0

I appreciate if you add support and update docs. Thanks!

[krrishdholakia]

it's already supported @yigitkonur

https://github.com/BerriAI/litellm/blob/9a5115d5ea6f7864ec3bd4809f2e0771bb90ebb5/model_prices_and_context_window.json#L4725

https://github.com/BerriAI/litellm/blob/9a5115d5ea6f7864ec3bd4809f2e0771bb90ebb5/litellm/llms/bedrock/chat/converse_handler.py#L43

[simonhir]

Is there already a way to add embedding models via azure_ai? In specific cohere.embed-multilingual-v3.0 Couldn't find anything in the documentation or code and the cohere provider seams to not allow setting the api_base.

[krrishdholakia]

cohere.embed-multilingual-v3.0

that model name looks like the bedrock one @simonhir

Re: azure ai embedding - not yet, can you create an issue, i'll add it tomorrow

[simonhir]

@krrishdholakia thanks for your fast answer. I created https://github.com/BerriAI/litellm/issues/5861. Hope that helps.

[AdithyanI]

@krrishdholakia Google just upgraded the gemini-1.5-pro. There is a new model gemini-1.5-pro-002 in GA. Also the flash. Can we support this please? We are specifically looking for in vertex_ai support. Thank you :)

https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-1.5-pro-002?hl=en&authuser=1&rapt=AEjHL4O-gl6qi8hhL9DBsAA1wb_wVoUUEkXTkJeARCSgbGPb03zHz3AwlLM98ApP59b3JpuRF68lhNwehNMo6lq0CFkU4lc4AyjAXa-eVDAGfubP1PeTUP4&project=win-compute-407516

[jbellis]

Support for gemini embedding models, please. (text-embedding-004 is free!)

[ClancyDennis]

Are there any plans to add the /image/edits endpoint currently supported by Dalle-2?

[krrishdholakia]

Google just upgraded the gemini-1.5-pro. There is a new model gemini-1.5-pro-002 in GA. Also the flash. Can we support this please?

done - https://github.com/BerriAI/litellm/blob/e19bb55e3b4c6a858b6e364302ebbf6633a51de5/model_prices_and_context_window.json#L2032

Support for gemini embedding models, please. (text-embedding-004 is free!)

@jbellis can you create an issue for this? it would help to know if you're v0 is for google ai studio or vertex ai

Are there any plans to add the /image/edits endpoint currently supported by Dalle-2?

@ClancyDennis didn't realize that, please create an issue, so we can track this.

[FireballDWF]

Related to Add Amazon Titan Text Premier #3774

[igorlima]

Hey @jbellis and @krrishdholakia!

• Support for gemini embedding models, please. (text-embedding-004 is free!) ⁺

• can you create an issue for this? ⁺

Great news! litellm already has built-in support for Gemini embeddings, though it wasn't highlighted in the documentation until now. 🎉

I've just proposed that with the PR below, which adds Gemini to the documentation page's list of supported embeddings.

• Add Gemini Embeddings Support

code snippet

Here's a quick and easy code snippet to show you how to use Gemini embeddings with litellm:

import os
os.environ["GEMINI_API_KEY"] = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

from litellm import embedding
response = embedding(
  model="gemini/text-embedding-004",
  input=["good morning from litellm"],
)
print(response)

[KamilFatkhiev]

Hi @krrishdholakia, are you considering adding support for yandexgpt and gigachat family models?

[krrishdholakia]

Hey @KamilFatkhiev the api's look openai compatible - https://developers.sber.ru/docs/ru/gigachat/api/reference/rest/post-chat

so should already work - https://docs.litellm.ai/docs/providers/openai_compatible

[KamilFatkhiev]

Hey @KamilFatkhiev the api's look openai compatible - https://developers.sber.ru/docs/ru/gigachat/api/reference/rest/post-chat

so should already work - https://docs.litellm.ai/docs/providers/openai_compatible

Oh, cool! What about yandexgpt? Are the differences too big? https://yandex.cloud/en-ru/docs/foundation-models/text-generation/api-ref/TextGeneration/completion

[krrishdholakia]

@KamilFatkhiev i don't have access to it - but if there's a mock endpoint we can use for testing it should be possible to add it

[guinmoon]

@KamilFatkhiev i don't have access to it - but if there's a mock endpoint we can use for testing it should be possible to add it

Hi. Thanks for a very cool project, I use it all the time! My fork has support for YandexGPT, but I'm afraid I made it so long ago that I can't create a pull request. My fork.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.