openai_api_compatible stream mode delimiter error

[bowenliang123]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to file this report (我已阅读并同意 Language Policy).

Dify version

0.5.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. set the stream delimiter to '\n\r' for custom qwen 千问 72b model with openai_api_compatible provider
2. llm chat failed with (red block in the screenshot) Error: can only concatenate str (not "NoneType") to str

✔️ Expected Behavior

❌ Actual Behavior

During steamlingly displaying the reply:

When the reply ends with error:

Code:

[dosubot[bot]]

🤖

Hey there, @bowenliang123! Great to see you back here again. Let's get this sorted out for you.

Based on the error message you're encountering, it seems like the system is trying to concatenate a string with a None value, which is not allowed in Python. This might be happening when the system is trying to concatenate the response from the model (delta.text or delta.delta.content) with the existing full text or full assistant content. If the model's response is None and it's being concatenated with the existing text, this error would occur.

Here's the relevant code:

full_text += text

full_assistant_content += delta.delta.content if delta.delta.content else ''

To fix this issue, you could modify the code to ensure that you're not trying to concatenate None with a string. You could do this by converting None to an empty string before concatenation. Here's how you could modify the code:

full_text += text if text is not None else ''

full_assistant_content += delta.delta.content if delta.delta.content is not None else ''

This is a general suggestion based on the provided code and the error message. The actual issue might be different based on the complete context and the specific conditions when the error occurs. If the error persists after trying this suggestion, please provide more details about the specific conditions when the error occurs.

Regarding your question about the function of the stream delimiter in the Dify codebase, the stream parameter in the _invoke method of the OAIAPICompatLargeLanguageModel class determines whether the response from the model should be streamed or not. If stream is set to True, the response from the model will be streamed, and the _handle_generate_stream_response method will be used to handle the response.

The stream_mode_delimiter in the credentials dictionary is used as a delimiter for the streamed response. This delimiter is used to split the streamed response into chunks. Each chunk is then processed separately.

Here is the relevant code:

# delimiter for stream response, need unicode_escape
import codecs
delimiter = credentials.get("stream_mode_delimiter", "\n\n")
delimiter = codecs.decode(delimiter, "unicode_escape")
for chunk in response.iter_lines(decode_unicode=True, delimiter=delimiter):
    if chunk:
        decoded_chunk = chunk.strip().lstrip('data: ').lstrip()
        chunk_json = None
        try:
            chunk_json = json.loads(decoded_chunk)
        # stream ended
        except json.JSONDecodeError as e:
            logger.error(f"decoded_chunk error,delimiter={delimiter},decoded_chunk={decoded_chunk}")
            yield create_final_llm_result_chunk(
                index=chunk_index + 1,
                message=AssistantPromptMessage(content=""),
                finish_reason="Non-JSON encountered."
            )
            break
        if not chunk_json or len(chunk_json['choices']) == 0:
            continue

        choice = chunk_json['choices'][0]
        finish_reason = chunk_json['choices'][0].get('finish_reason')
        chunk_index += 1

In this code, the stream_mode_delimiter is retrieved from the credentials dictionary, with a default value of "\n\n" if it is not provided. The codecs.decode function is used to decode the delimiter from a unicode escape sequence to a string. The response.iter_lines function is then used to iterate over the response in chunks, with each chunk being separated by the delimiter. Each chunk is then processed separately.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• api/core/model_runtime/model_providers/openai/llm/llm.py
• api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[bowenliang123]

The orginal feature and bug introduced by https://github.com/langgenius/dify/pull/2190. I don't know the reason that the guard conditions in purple block does not work as expectedly and I suggested a temporary workaround that works for me in blue block, for reference.

cc @geosmart @takatost