After enabling Streaming Mode as Asynchronous Modified Filter in Azure OpenAI, the returned results will throw an error.

[sagarey]

Self Checks

• [X] This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.2

Cloud or Self Hosted

Cloud, Self Hosted (Docker)

Steps to reproduce

Azure OpenAI's default content filter affects streaming response, returning data in paragraphs rather than by characters. Therefore, I adjusted the streaming mode in the custom content filter feature of Azure AI Studio to Asynchronous Modified Filter, make the streaming response work as expected, which triggered this bug.

✔️ Expected Behavior

❌ Actual Behavior

Debugging page and the app publishing both have issues. The chatbox streams the results as expected, but finally throws an error and clears the frontend display. I reproduced this problem in local Docker

[dosubot[bot]]

To address the error {azure_openai} Error: 'NoneType' object has no attribute 'content' you're encountering, it's crucial to implement checks for None before accessing the content attribute in your streaming response handling code. This error typically occurs when a response chunk is missing expected data, leading to attempts to access attributes of a None object.

Here's a quick fix you can apply to your response handling method, such as _handle_chat_generate_stream_response, to safely check for None:

def _handle_chat_generate_stream_response(self, model: str, credentials: dict,
                                          response: Stream[ChatCompletionChunk],
                                          prompt_messages: list[PromptMessage],
                                          tools: Optional[list[PromptMessageTool]] = None) -> Generator:
    for chunk in response:
        if chunk.choices and chunk.choices[0].delta and chunk.choices[0].delta.delta is not None:
            # Now safe to access delta.delta.content and other attributes
            # Your existing logic here...

This modification ensures your code skips over chunks where delta.delta is None, preventing the NoneType object has no attribute 'content' error.

For handling streaming responses, especially when using the "Asynchronous Modified Filter" mode, it's important to process each response chunk as it arrives, considering that some chunks might not contain the data structure you expect. Implementing the above check at every point where you access attributes of potentially None objects in the streaming context will help avoid similar errors.

If you continue to face issues or have more questions, feel free to ask for further assistance.

Sources

• api/core/model_runtime/model_providers/azure_openai/llm/llm.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[sagarey]

To address the error {azure_openai} Error: 'NoneType' object has no attribute 'content' you're encountering, it's crucial to implement checks for None before accessing the content attribute in your streaming response handling code. This error typically occurs when a response chunk is missing expected data, leading to attempts to access attributes of a None object.

Here's a quick fix you can apply to your response handling method, such as _handle_chat_generate_stream_response, to safely check for None:

def _handle_chat_generate_stream_response(self, model: str, credentials: dict,
                                          response: Stream[ChatCompletionChunk],
                                          prompt_messages: list[PromptMessage],
                                          tools: Optional[list[PromptMessageTool]] = None) -> Generator:
    for chunk in response:
        if chunk.choices and chunk.choices[0].delta and chunk.choices[0].delta.delta is not None:
            # Now safe to access delta.delta.content and other attributes
            # Your existing logic here...

This modification ensures your code skips over chunks where delta.delta is None, preventing the NoneType object has no attribute 'content' error.

For handling streaming responses, especially when using the "Asynchronous Modified Filter" mode, it's important to process each response chunk as it arrives, considering that some chunks might not contain the data structure you expect. Implementing the above check at every point where you access attributes of potentially None objects in the streaming context will help avoid similar errors.

If you continue to face issues or have more questions, feel free to ask for further assistance.

Sources

• api/core/model_runtime/model_providers/azure_openai/llm/llm.py

About Dosu

Yeah, also located the error source. I'll make a pr.

[dosubot[bot]]

It sounds like you've pinpointed the source of the error and are planning to address it with a pull request. That's great to hear! If you need any further assistance or have additional questions as you work on the fix, feel free to reach out. Good luck with your PR!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.