litellm
litellm copied to clipboard
Bedrock pixtral fix
Title
Fix support for Pixtral Large (25.02) from Amazon Bedrock provider.
Relevant issues
Fixes #10360
Pre-Submission checklist
- [ x] I have added testing in the
tests/litellm/directory, Adding at least 1 test is a hard requirement - see details - [ x] I have added a screenshot of my new test passing locally
- [ x] My PR passes all unit tests on
make test-unit - [ x] My PR's scope is as isolated as possible; It only solves 1 specific problem
Type
🐛 Bug Fix
Changes
'transform_response' in 'base_invoke_transformation.py' expected Mistral models from Bedrock to produce responses with the 'outputs' key; However, Pixtral Large (25.02) responds in a different structure with the 'choices' key. The changes made aim to accommodate this variation:
- Added a new "get_outputText" static method to "AmazonMistralConfig". This method extracts the string response from the LLM differently based on the structure of the model response. If the response holds a "choices" key, that will take priority; Otherwise, it will search for the "outputs" key. If no adequate keys are found, a Bedrock error will be raised.
- Replaced the Mistral route for 'transform_response' in 'base_invoke_transformation.py' with this static method.
- Added a new test file under "/bedrock/chat" to test this static method.
The reasoning behind abstracting this logic behind a static method is to facilitate unit testing in an isolated manner away from the rest of the "transform_response" method. The "AmazonMistralConfig" class was chosen to follow a similar pattern established by the use of "AmazonInvokeNovaConfig" and "AmazonAnthropicClaude3Config" within the method.
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| litellm | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Apr 30, 2025 2:47pm |
Hello guys, is there any blocking point to include this patch in the next release? Thanks :)
Great pr - thanks for the work on this @ajac-zero