got invalid_request_error when use gpt-4-vision-preview

[infometa]

Self Checks

• [x] This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.5.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Create an Assistant with gpt-4-vision-preview model and knowledgebase . after upload the image got and send the message got the error message {'error': {'message': 'Invalid chat format. Unexpected keys in messages.', 'type': 'invalid_request_error', 'param': None, 'code': None}\

api-1       | INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 400 Bad Request"
api-1       | ERROR:core.application_manager:Unknown Error when generating
api-1       | Traceback (most recent call last):
api-1       |   File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 96, in invoke
api-1       |     result = self._invoke(model, credentials, prompt_messages, model_parameters, tools, stop, stream, user)
api-1       |   File "/app/api/core/model_runtime/model_providers/openai/llm/llm.py", line 74, in _invoke
api-1       |     return self._chat_generate(
api-1       |   File "/app/api/core/model_runtime/model_providers/openai/llm/llm.py", line 551, in _chat_generate
api-1       |     response = client.chat.completions.create(
api-1       |   File "/usr/local/lib/python3.10/site-packages/openai/_utils/_utils.py", line 275, in wrapper
api-1       |     return func(*args, **kwargs)
api-1       |   File "/usr/local/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 667, in create
api-1       |     return self._post(
api-1       |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1208, in post
api-1       |     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
api-1       |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 897, in request
api-1       |     return self._request(
api-1       |   File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 988, in _request
api-1       |     raise self._make_status_error_from_response(err.response) from None
api-1       | openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid chat format. Unexpected keys in messages.', 'type': 'invalid_request_error', 'param': None, 'code': None}}
api-1       | 
api-1       | During handling of the above exception, another exception occurred:
api-1       | 
api-1       | Traceback (most recent call last):
api-1       |   File "/app/api/core/application_manager.py", line 173, in _generate_worker
api-1       |     runner.run(
api-1       |   File "/app/api/core/app_runner/assistant_app_runner.py", line 248, in run
api-1       |     self._handle_invoke_result(
api-1       |   File "/app/api/core/app_runner/app_runner.py", line 230, in _handle_invoke_result
api-1       |     self._handle_invoke_result_stream(
api-1       |   File "/app/api/core/app_runner/app_runner.py", line 263, in _handle_invoke_result_stream
api-1       |     for result in invoke_result:
api-1       |   File "/app/api/core/features/assistant_cot_runner.py", line 137, in run
api-1       |     chunks: Generator[LLMResultChunk, None, None] = model_instance.invoke_llm(
api-1       |   File "/app/api/core/model_manager.py", line 70, in invoke_llm
api-1       |     return self.model_type_instance.invoke(
api-1       |   File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 111, in invoke
api-1       |     raise self._transform_invoke_error(e)
api-1       | core.model_runtime.errors.invoke.InvokeBadRequestError: [openai] Bad Request Error, Error code: 400 - {'error': {'message': 'Invalid chat format. Unexpected keys in messages.', 'type': 'invalid_request_error', 'param': None, 'code': None}}

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot[bot]]

Based on the error message you've shared, it seems like the request being sent to the gpt-4-vision-preview model doesn't match the expected format. This could be due to unexpected keys in the messages or parameters that don't adhere to the model's configuration.

Here are some steps you can take to resolve this issue:

1. Validate Message Format: The gpt-4-vision-preview model operates in chat mode and expects a specific message format. If you're using the response_format parameter and it's set to json_object, make sure your message structure complies with the JSON object format expected by the model.

2. Check Parameter Usage: The error suggests that there might be unexpected or unsupported parameters in the request. You should verify that all parameters sent in the request match those defined in the parameter_rules of the model's YAML configuration file. You can find the YAML configuration for the gpt-4-vision-preview model in the Dify codebase here.

3. Adhere to Parameter Constraints: The values for parameters like max_tokens, temperature, top_p, presence_penalty, and frequency_penalty should fall within the specified ranges or match the expected data types. For max_tokens, ensure the value is between 1 and 4096.

4. Optional Parameters: If you're using the seed parameter for deterministic outputs, ensure it is an integer. However, this parameter is not required and can be omitted if determinism is not a concern.

5. Request Format: If your request includes an image along with text (since the model supports vision features), ensure the format of the request properly encapsulates both the image data and the text in a manner expected by the model. This might require encoding the image in a specific way and structuring the request payload to include both elements distinctly.

6. Debugging and Logs: Utilize any debugging tools or logs available in your self-hosted Dify setup to capture more details about the request and the exact nature of the format error. This can provide clues on what part of the request is malformed.

By carefully reviewing and adjusting the request payload to match the model's expected parameters and formats as outlined in the YAML configuration, you should be able to resolve the BadRequestError with a 400 error code due to an invalid chat format with unexpected keys in messages.

If you continue to encounter issues, please provide more details about the request payload you're sending to the model, and I'll be happy to help further.

Sources

• api/core/model_runtime/model_providers/openai/llm/gpt-4-vision-preview.yaml
• api/core/model_runtime/model_providers/__base/large_language_model.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[crazywoola]

Please upgrade to latest version to see if this issue persists. :)