litellm icon indicating copy to clipboard operation
litellm copied to clipboard

Bedrock pixtral fix

Open ajac-zero opened this issue 6 months ago • 3 comments

Title

Fix support for Pixtral Large (25.02) from Amazon Bedrock provider.

Relevant issues

Fixes #10360

Pre-Submission checklist

  • [ x] I have added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • [ x] I have added a screenshot of my new test passing locally Screenshot 2025-04-30 at 8 27 15 a m
  • [ x] My PR passes all unit tests on make test-unit
  • [ x] My PR's scope is as isolated as possible; It only solves 1 specific problem

Type

🐛 Bug Fix

Changes

'transform_response' in 'base_invoke_transformation.py' expected Mistral models from Bedrock to produce responses with the 'outputs' key; However, Pixtral Large (25.02) responds in a different structure with the 'choices' key. The changes made aim to accommodate this variation:

  1. Added a new "get_outputText" static method to "AmazonMistralConfig". This method extracts the string response from the LLM differently based on the structure of the model response. If the response holds a "choices" key, that will take priority; Otherwise, it will search for the "outputs" key. If no adequate keys are found, a Bedrock error will be raised.
  2. Replaced the Mistral route for 'transform_response' in 'base_invoke_transformation.py' with this static method.
  3. Added a new test file under "/bedrock/chat" to test this static method.

The reasoning behind abstracting this logic behind a static method is to facilitate unit testing in an isolated manner away from the rest of the "transform_response" method. The "AmazonMistralConfig" class was chosen to follow a similar pattern established by the use of "AmazonInvokeNovaConfig" and "AmazonAnthropicClaude3Config" within the method.

ajac-zero avatar Apr 30 '25 14:04 ajac-zero

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
litellm ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 30, 2025 2:47pm

vercel[bot] avatar Apr 30 '25 14:04 vercel[bot]

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Apr 30 '25 14:04 CLAassistant

Hello guys, is there any blocking point to include this patch in the next release? Thanks :)

Extremys avatar May 13 '25 09:05 Extremys

Great pr - thanks for the work on this @ajac-zero

krrishdholakia avatar May 15 '25 06:05 krrishdholakia