OpenAI Plugin Fails to Send Multimodal Messages with Images

[leozanee]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Title

OpenAI Plugin Fails to Send Multimodal Messages with Images

Description

The plugin exhibits inconsistent behavior in an unknown scenario, causing the OpenAI plugin to malfunction and fail to send multimodal messages containing images.

Steps to Reproduce

1. Use the OpenAI plugin (versions 0.0.8 - 0.0.13).
2. Attempt to send a multimodal message with an image (e.g., content including both text and an image).
3. Check the logs and observe the error.

Error Logs

When sending a multimodal message with an image, the following error appears in the logs:

api-1            | /app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
api-1            |   PydanticSerializationUnexpectedValue: Expected `str` but got `list` with value `[TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)]` - serialized value may not be as expected
api-1            |   PydanticSerializationUnexpectedValue: Expected `PromptMessageContent` but got `TextPromptMessageContent` with value `TextPromptMessageContent(...ata='can you see this?')` - serialized value may not be as expected
api-1            |   return self.__pydantic_serializer__.to_python(
plugin_daemon-1  | [GIN] 2025/03/25 - 07:07:21 | 200 |    9.872525ms |      172.19.0.9 | POST     "/plugin/3559813e-c1d2-4fdc-ada2-e05ac28e4f45/dispatch/llm/num_tokens"
api-1            | 2025-03-25 07:07:21.159 ERROR [Thread-3 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating
api-1            | Traceback (most recent call last):
api-1            |   File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker
api-1            |     runner.run(
api-1            |   File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run
api-1            |     self.get_pre_calculate_rest_tokens(
api-1            |   File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens
api-1            |     prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages)
api-1            |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens
api-1            |     self._round_robin_invoke(
api-1            |   File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
api-1            |     return function(*args, **kwargs)
api-1            |            ^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens
api-1            |     return plugin_model_manager.get_llm_num_tokens(
api-1            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens
api-1            |     for resp in response:
api-1            |                 ^^^^^^^^
api-1            |   File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
api-1            |     self._handle_plugin_daemon_error(error.error_type, error.message)
api-1            |   File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
api-1            |     raise PluginInvokeError(description=message)
api-1            | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}
api-1            | 2025-03-25 07:07:21.169 ERROR [Dummy-2] [base_app_generate_response_converter.py:123] - {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}

Test Results

• Tested the following third-party provider plugins:
  • Gemini-0.0.8: Works normally, no errors.
  • Anthropic-0.0.12: Works normally, no errors.
  • OpenAI (0.0.8 - 0.0.13): Errors occur, only with the OpenAI plugin.
• In Dify Cloud-Hosted Environment:
  • Upgrading the OpenAI plugin to 0.0.13 reproduces the issue.
  • Rolling back to 0.0.8 resolves the issue, and it works normally.
• In Local Docker-Hosted Environment:
  • Even rolling back to 0.0.8 does not resolve the issue; the error persists.

Analysis

• Suspect that changes in the plugin runtime environment are causing the issue.
• The problem was initially encountered while developing a new third-party provider plugin:
  • Works fine in debug mode.
  • After packaging, the same error occurs.
• Reviewed the source code of the OpenAI and Anthropic plugins but found no significant logical differences.
• The root cause of the error is unclear and may be related to Pydantic’s serialization validation logic or compatibility issues with the plugin environment.

Request for Assistance

Since the OpenAI plugin is an official implementation and more representative, but the cause of the issue remains elusive, I’m seeking help to identify and resolve the problem. Thank you!

✔️ Expected Behavior

No response

❌ Actual Behavior

No response