[BUG] Content Filter errors originating from LLM steps contain a mix of JSON and text representations of Python objects

[avante-nick]

Describe the bug When an LLM step in a PromptFlow run generates an error, it produces an error message format that is very difficult to consume with code. For example:

{ "error": { "code": "UserError", "message": "OpenAI API hits BadRequestError: Error code: 400 - {'error': {'message': \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': True, 'severity': 'medium'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}} [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]" } }

In this specific scenario, I'd want to have an error handler that could identify that the error was related to content filtering, and be able to do different things depending on which filter was hit, and to what degree. This is difficult with the current error, however, because it's a JSON object containing a string object which itself seems to contain a string representation of a Python object.

How To Reproduce the bug

1. Set up a basic Prompt Flow with an LLM step, and the default Content Filter policy
2. Send input to the flow that triggers the content filter policy

Expected behavior The PromptFlow should return an error that is fully JSON formatted, not a mix of JSON and a text representation of a Python object

Running Information(please complete the following information):

• Promptflow Package Version: Using the hosted preview on ai.azure.com

[github-actions[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

[avante-nick]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

@zzn2 do you know why this issue was closed due to no recent activity? I did not see any requests for further information. Happy to provide that if needed

[thy09]

@avante-nick The issue is closed because of a github action, we will adjust the rule. As for your ask, it is a valid ask and we are planning it as a feature.