[Bug]: ContextWindowExceeded error

[neubig]

Is there an existing issue for the same bug?

• [X] I have checked the existing issues.

Describe the bug and reproduction steps

Twice so far I have seen the following context window exceeded error:

litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'litellm.BadRequestError: BedrockException - {"message":"messages.32.content: Conversation blocks and tool result blocks cannot be provided in the same turn."}\n. Enable \'litellm.modify_params=True\' (for PROXY do: `litellm_settings::modify_params: True`) to insert a dummy assistant message and fix this error.\nReceived Model Group=claude-3-5-sonnet-20241022\nAvailable Model Group Fallbacks=[\'anthropic/claude-3-5-sonnet-20241022\']\nError doing the fallback: litellm.BadRequestError: litellm.ContextWindowExceededError: AnthropicError - {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 209303 tokens > 200000 maximum"}}\nReceived Model Group=anthropic/claude-3-5-sonnet-20241022\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: litellm.ContextWindowExceededError: AnthropicError - {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 209303 tokens > 200000 maximum"}}', 'type': None, 'param': None, 'code': '400'}}

Here is the full trace of an interaction that caused the error: https://www.all-hands.dev/share?share_id=5bae6fff47ffcfbcd1d64eceb3318822d528c555ba6e004381dde4a7813046ac

This is using app.all-hands.dev and the All Hands litellm proxy for serving the models. I though that this error would have been fixed by #4977 , but it seems that the problem is not entirely fixed. I don't have visibility into what causes the error, but it's probably better to pin it here.

(CC: @enyst and @csmith49)

OpenHands Installation

app.all-hands.dev

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

No response

[enyst]

I guess our code received a BadRequestError in this case, from the proxy, instead of the underlying ContextWindowExceededError. 👀

[neubig]

Ah, @xingyaoww , any ideas how we could possibly debug this?

[xingyaoww]

I think maybe we should start an issue in LiteLLM repo, since LiteLLM should map this to ContextWindowExceededError instead of BadRequestError?

[neubig]

OK, sounds good, will do.

[neubig]

There's already an issue open: https://github.com/BerriAI/litellm/issues/6629

[enyst]

Oh, great find! I can fix it meanwhile for us to capture that BadRequestError.

[neubig]

OpenHands is already working on it btw.

[neubig]

PR sent upstream: https://github.com/BerriAI/litellm/pull/6973

[enyst]

I'm not sure that's it, because it converts an exception to RateLimitError. That may be similar problem, but we need ContextWindowExceededError mapped from a dozen lines above, probably:

• https://github.com/BerriAI/litellm/pull/6973/files#diff-b95f3e294fcac9a4b8777e7f03c10d94feffb6ac49a34f233bb2ead2710c29cbL736

That is, it can add to this code:

elif custom_llm_provider == "bedrock":
                if (
                    "too many tokens" in error_str
                    or "expected maxLength:" in error_str
                    or "Input is too long" in error_str
                    or "prompt: length: 1.." in error_str
                    or "Too many input tokens" in error_str
                ):
                    exception_mapping_worked = True
                    raise ContextWindowExceededError(
                        message=f"BedrockException: Context Window Error - {error_str}",
                        model=model,
                        llm_provider="bedrock",
                    )

The Anthropic message, which according to the error in the original post here, is:

"prompt is too long:"

[neubig]

Oh, thanks so much. I'll send a separate PR.

[neubig]

OK, here's the open PR: https://github.com/BerriAI/litellm/pull/6975

[neubig]

This should be fixed upstream: https://github.com/BerriAI/litellm/pull/6994

@xingyaoww , when the new release is made could you re-deploy the proxy?

[neubig]

Hey @xingyaoww , I'm still experiencing this with the most recent version of the proxy, I just saw it now:

litellm.BadRequestError: OpenAIException - Error code: 500 - {'error': {'message': 'litellm.APIConnectionError: BedrockException - {"message":"You do not have access to explicit prompt caching"}\nReceived Model Group=claude-3-5-sonnet-20241022\nAvailable Model Group Fallbacks=['anthropic/claude-3-5-sonnet-20241022']\nError doing the fallback: litellm.BadRequestError: litellm.ContextWindowExceededError: AnthropicError - {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 204935 tokens > 200000 maximum"}}\nReceived Model Group=anthropic/claude-3-5-sonnet-20241022\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: litellm.ContextWindowExceededError: AnthropicError - {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 204935 tokens > 200000 maximum"}}', 'type': None, 'param': None, 'code': '500'}}