gateway
gateway copied to clipboard
Portkey-AI
AWS Bedrock Validation Exception Not Handled.
What Happened?
When uploading an image through Portkey, the outgoing request to Bedrock API returned a 200 with a "application/vnd.amazon.eventstream". However, the stream's sole message was exception{"message":"The model returned the following errors: Could not process image"}. Thus, a fallback was not triggered even though no data (besides the error) was able to be streamed.
What Should Have Happened?
There should be handling of these errors or a way to surface them to applications using Portkey.
Relevant Code Snippet
URL being accessed by Portkey: https://bedrock-runtime.us-east-1.amazonaws.com/model/us.anthropic.claude-3-5-sonnet-20241022-v2:0/converse-stream
Your Twitter/LinkedIn
No response
Hey @jczstudios openai streams do not return error chunks, so there are no handlers for error events in the transformers. It would be nice to have a unified chunk signature for streaming error chunks though. @VisargD what do you think?
Hi! I think most of the validation error chunks for stream are sent as the first chunk. Maybe Gateway can wait for the first chunk to decide whether its a 200 or a validation error.
There are chances that the error chunk is returned after the first chunk and for that we will need a unified error chunk. But for validation error errors, I think we can simply solve it by waiting for the first chunk and then taking the decision.
makes sense, but we can handle both the cases the same way if we have a unified signature for error chunks @VisargD
I looked into making a PR for this myself but I noticed the existing architecture doesn't support throwing errors outside of the initialization code, is this correct?
I also contacted AWS to ask about this, their documentation here: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html#API_runtime_ConverseStream_ResponseSyntax states that a validationError returns a 400 error code but that is obviously not happening. If they can fix this on their end no change would be needed but perhaps I'm not understanding something correctly with their documentation.
Is there any chance a plugin could help fix this behavior on our end?
@jczstudios @Galkon sorry I missed the thread,
One way I can think of is to consume the error inside the gateway and end the stream, but that would not help with surfacing the error for a consumer
The other way is to return a standardized error chunk (we can add a new field in this format https://platform.openai.com/docs/api-reference/chat-streaming/streaming), so that even if the HTTP code is 200, consumers can handle the stream accordingly
I don't see any other way tbh, and honestly I'm open to any suggestions on the matter
I think passing it into the stream would be fine, would give me a way to present something to the user :D
I think passing it into the stream would be fine, would give me a way to present something to the user :D
Got it, that does appear to be the best way to do it. I will pick it up if I have some time on my hands, please feel free to add more information or contributions for this