dify icon indicating copy to clipboard operation
dify copied to clipboard
Published 3 days ago • verified publisher icon langgenius

1.1.3 Unable to use multimodal, upload pictures for chat

Open lichao4Java opened this issue 9 months ago • 19 comments

Self Checks

  • [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
  • [x] I have searched for existing issues search for existing issues, including closed ones.
  • [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

PydanticSerializationUnexpectedValue: Expected str but got list with value [TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)] - serialized value may not be as expected PydanticSerializationUnexpectedValue: Expected PromptMessageContent but got TextPromptMessageContent with value TextPromptMessageContent(...'>, data='这是什么') - serialized value may not be as expected

✔️ Expected Behavior

can upload pictures for chat

❌ Actual Behavior

The GPT-4o claude3-7 model selected by OpenAI and Openrouter cannot upload images for chat, and the log error message is as follows:

025-03-26 06:29:04.469 ERROR [Thread-387 (_generate_worker)] [app_generator.py:260] - Unknown Error when generating
Traceback (most recent call last):
  File "/app/api/core/app/apps/agent_chat/app_generator.py", line 240, in _generate_worker
    runner.run(
  File "/app/api/core/app/apps/agent_chat/app_runner.py", line 61, in run
    self.get_pre_calculate_rest_tokens(
  File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens
    prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens
    self._round_robin_invoke(
  File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens
    return plugin_model_manager.get_llm_num_tokens(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens
    for resp in response:
                ^^^^^^^^
  File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
    self._handle_plugin_daemon_error(error.error_type, error.message)
  File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
    raise PluginInvokeError(description=message)
core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}
2025-03-26 06:29:04.474 ERROR [Dummy-388] [base_app_generate_response_converter.py:123] - {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}

lichao4Java avatar Mar 26 '25 06:03 lichao4Java

The error you're encountering seems to be related to the expected data type for prompt_messages.content.type. In the Dify codebase, this should be of type PromptMessageContentType, which can be either PromptMessageContentType.TEXT or PromptMessageContentType.IMAGE [1].

To resolve this issue, ensure that the prompt_messages.content.type is correctly set to one of these expected types. If you're trying to upload images, make sure that the type is set to PromptMessageContentType.IMAGE.

Additionally, for handling image uploads in multimodal chat scenarios, ensure that you have configured the necessary settings for file uploads, as the standalone image upload feature has been replaced with a comprehensive file upload capability [2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

dosubot[bot] avatar Mar 26 '25 06:03 dosubot[bot]

What's your plugin version?

crazywoola avatar Mar 26 '25 06:03 crazywoola

What's your plugin version?

OpenAI 0.0.13 OpenRouter 0.05

Looks like they are all up to date, I just upgraded to version 1.1.3

lichao4Java avatar Mar 26 '25 06:03 lichao4Java

What's your plugin version? @crazywoola I found that in cloud.dify.ai 1.1.3, OpenAI plugin version 0.0.7. Uploading images using GPT-4o mini did not generate any errors, but it also failed to recognize the images.

Is the latest plugin reporting an error when uploading images a bug?

Image

lichao4Java avatar Mar 26 '25 07:03 lichao4Java

4o

Image

4o-mini

Image

I can not reproduce this error.

crazywoola avatar Mar 26 '25 07:03 crazywoola

Image

When I input the image URL, I get an error:

{ "code": "invalid_param", "message": "1 validation error for File\n Value error, Invalid file url [type=value_error, input_value={'id': None, 'tenant_id':...y__file__', 'url': None}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.9/v/value_error", "status": 400 }

Image

lichao4Java avatar Mar 26 '25 08:03 lichao4Java

Ahh, I see, it should be a problem of the openrouter. https://github.com/langgenius/dify-official-plugins/pulls?q=is%3Apr+openrouter+is%3Aclosed

crazywoola avatar Mar 26 '25 08:03 crazywoola

Ahh, I see, it should be a problem of the openrouter. https://github.com/langgenius/dify-official-plugins/pulls?q=is%3Apr+openrouter+is%3Aclosed

openai and openrouter model, The input URL and the last image cannot have a conversation, although these two errors are different.

Has it been fixed?

this is openrouter version: Image

openai version:

Image

lichao4Java avatar Mar 26 '25 09:03 lichao4Java

Not yet, we haven't got time to fix this yet.

crazywoola avatar Mar 26 '25 14:03 crazywoola

https://github.com/langgenius/dify/issues/16353#issuecomment-2754785232

kurokobo avatar Mar 26 '25 15:03 kurokobo

Temporary workarounds have been released. For those facing this issue, upgrade your plugin and give it another try :)

  • Azure OpenAI: >= 0.0.11
  • OpenAI: >= 0.0.14
  • OpenRouter: >=0.0.6

kurokobo avatar Mar 27 '25 01:03 kurokobo

临时解决方法已发布。对于遇到此问题的用户，请升级您的插件并再试一次 :)

  • Azure OpenAI：>= 0.0.11
  • OpenAI：> = 0.0.14
  • OpenRouter：>=0.0.6

OpenAI：= 0.0.14 OpenRouter：=0.0.6

It hasn't been released yet。

Image

where i can install?

lichao4Java avatar Mar 27 '25 08:03 lichao4Java

  • Azure OpenAI：>= 0.0.11
  • OpenAI：> = 0.0.14
  • OpenRouter：>=0.0.6

OpenAI：= 0.0.14 OpenRouter：=0.0.6

It hasn't been released yet。

Image

where i can install?

And OpenRouter is the same, 0.0.6 is not released yet.

sunisstar avatar Mar 27 '25 12:03 sunisstar

Oh, my bad. I just checked that the CI passed and that Azure OpenAI was released correctly, then post it.

@crazywoola @laipz8200 Is there any issue on the marketplace side regarding the OpenAI and OpenRouter plugins? My PRs are not released yet, could it be a CDN issue? I thought that once the GHA CI passes for the PRs, it would automatically be published as is.

kurokobo avatar Mar 27 '25 14:03 kurokobo

Oh, my bad. I just checked that the CI passed and that Azure OpenAI was released correctly, then post it.

@crazywoola @laipz8200 Is there any issue on the marketplace side regarding the OpenAI and OpenRouter plugins? My PRs are not released yet, could it be a CDN issue? I thought that once the GHA CI passes for the PRs, it would automatically be published as is.

At this stage, the issue has been identified as belonging to the dify_plugin SDK. The official team has not yet pushed the SDK to PyPI. For now, plugins encountering issues, you can manually modify the requirements.txt file in the plugin by changing: dify_plugin~=xxx to: dify_plugin~=xxx,<0.0.1b74 Then resolve it by installing locally.

VLLM plugin and OpenAI-API-compatible plugin still error :

Run failed: [vllm] Error: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelInvokeLLMRequest\nprompt_messages.content.type\n Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n For further information visit [https://errors.pydantic.dev/2.10/v/literal_error"}](https://errors.pydantic.dev/2.10/v/literal_error%22%7D)

Run failed: [openai_api_compatible] Error: PluginInvokeError: {"args":{"description":"[models] Error: API request failed with status code 500: Internal Server Error"},"error_type":"InvokeError","message":"[models] Error: API request failed with status code 500: Internal Server Error"}

moshilangzi avatar Mar 27 '25 18:03 moshilangzi

Problem fixed, now the plugins with workaround available on the marketplace 😃

  • Azure OpenAI：>= 0.0.11
  • OpenAI：> = 0.0.14
  • OpenRouter：>=0.0.6

@Yeuoly Thanks for your rapid assistance!

kurokobo avatar Mar 28 '25 03:03 kurokobo

Similar Problem

Image

Version Information:

  1. dify-plugin-daemon:0.0.6-local
  2. dify-api: 1.1.1
  3. Bedrock plugin: have tried 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9
  4. Bedrock Claude 3.5 / Claude 3.5 V2 / Claude 3.7

Log below:

2025-03-28 05:49:33,203.203 DEBUG [Thread-398 (_generate_worker)] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/a1fdafda-0586-468a-9943-22226bb07ee6/dispatch/llm/num_tokens HTTP/1.1" 200 None
2025-03-28 05:49:33,205.205 ERROR [Thread-398 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating
Traceback (most recent call last):
  File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker
    runner.run(
  File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run
    self.get_pre_calculate_rest_tokens(
  File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens
    prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens
    self._round_robin_invoke(
  File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens
    return plugin_model_manager.get_llm_num_tokens(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens
    for resp in response:
                ^^^^^^^^
  File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
    self._handle_plugin_daemon_error(error.error_type, error.message)
  File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
    raise PluginInvokeError(description=message)
core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/literal_error"}

X4tar avatar Mar 28 '25 08:03 X4tar

upgrade dify pluins

发件人: X4tar @.> 发送时间: 2025年3月28日 8:14 收件人: langgenius/dify @.> 抄送: chris_lee @.>; Author @.> 主题: Re: [langgenius/dify] 1.1.3 Unable to use multimodal, upload pictures for chat (Issue #16816)

Similar Problem

image.png (view on web)https://github.com/user-attachments/assets/d34c35d4-4649-4349-a622-e538a7dc20e5

Version Information:

  1. dify-plugin-daemon:0.0.6-local
  2. dify-api: 1.1.1
  3. Bedrock plugin: have tried 0.05
  4. Bedrock Claude 3.5 / Claude 3.5 V2 / Claude 3.7

Log below:

2025-03-28 05:49:33,203.203 DEBUG [Thread-398 (_generate_worker)] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/a1fdafda-0586-468a-9943-22226bb07ee6/dispatch/llm/num_tokens HTTP/1.1" 200 None 2025-03-28 05:49:33,205.205 ERROR [Thread-398 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating Traceback (most recent call last): File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker runner.run( File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run self.get_pre_calculate_rest_tokens( File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens self._round_robin_invoke( File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke return function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens return plugin_model_manager.get_llm_num_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens for resp in response: ^^^^^^^^ File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream self._handle_plugin_daemon_error(error.error_type, error.message) File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error raise PluginInvokeError(description=message) core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n For further information visit https://errors.pydantic.dev/2.11/v/literal_error"}

― Reply to this email directly, view it on GitHubhttps://github.com/langgenius/dify/issues/16816#issuecomment-2760513329, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADE6BES5HUT7HWMVDSPSM6L2WUAHXAVCNFSM6AAAAABZZN2S5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGUYTGMZSHE. You are receiving this because you authored the thread.Message ID: @.***>

[X4tar]X4tar left a comment (langgenius/dify#16816)https://github.com/langgenius/dify/issues/16816#issuecomment-2760513329 Similar Problem

image.png (view on web)https://github.com/user-attachments/assets/d34c35d4-4649-4349-a622-e538a7dc20e5

Version Information:

  1. dify-plugin-daemon:0.0.6-local
  2. dify-api: 1.1.1
  3. Bedrock plugin: have tried 0.05
  4. Bedrock Claude 3.5 / Claude 3.5 V2 / Claude 3.7

Log below:

2025-03-28 05:49:33,203.203 DEBUG [Thread-398 (_generate_worker)] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/a1fdafda-0586-468a-9943-22226bb07ee6/dispatch/llm/num_tokens HTTP/1.1" 200 None 2025-03-28 05:49:33,205.205 ERROR [Thread-398 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating Traceback (most recent call last): File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker runner.run( File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run self.get_pre_calculate_rest_tokens( File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens self._round_robin_invoke( File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke return function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens return plugin_model_manager.get_llm_num_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens for resp in response: ^^^^^^^^ File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream self._handle_plugin_daemon_error(error.error_type, error.message) File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error raise PluginInvokeError(description=message) core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n For further information visit https://errors.pydantic.dev/2.11/v/literal_error"}

― Reply to this email directly, view it on GitHubhttps://github.com/langgenius/dify/issues/16816#issuecomment-2760513329, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADE6BES5HUT7HWMVDSPSM6L2WUAHXAVCNFSM6AAAAABZZN2S5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGUYTGMZSHE. You are receiving this because you authored the thread.Message ID: @.***>

lichao4Java avatar Mar 28 '25 13:03 lichao4Java

upgrade dify pluins …

Thanks for you suggestion. I modifed the problem description, I have tried Bedrock plugin: have tried 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9 in my two different deployments.

X4tar avatar Mar 28 '25 13:03 X4tar