dify List out of range error using big prompt with sonnet 3.5

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

doesn't work with bigger prompts sometimes (60k characters)

✔️ Expected Behavior

llm response

❌ Actual Behavior

[anthropic] Error: list index out of range

Sep 10 '24 01:09 lucaseatp

@lucaseatp Hello there! I'm here to assist you with any bugs, questions, or contributions while you wait for a human maintainer. Let's tackle this issue together!

To resolve the "list index out of range" error when using large prompts (60k characters) with Sonnet 3.5 in Dify version 0.7.3, you can lower the value of "Max token" in the parameter settings of the Prompt Eng. This is mentioned in the FAQ section of the document:

### 9. How to solve the following error prompt?

```JSON
Query or prefix prompt is too long, you can reduce the preix prompt, or shrink the max token, or switch to a llm with a larger token limit size

You can lower the value of "Max token" in the parameter settings of the Prompt Eng.


By reducing the max token value, you can ensure that the length of the prompt does not exceed the acceptable range of the language model, thereby avoiding the "list index out of range" error. According to the model configuration for Claude 3.5 Sonnet, the `max_tokens` parameter has a default value of 4096 and a maximum limit of 4096. Adjusting this parameter within the allowed range can help manage the prompt size effectively <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/application-orchestrate/llms-use-faq.md)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openrouter/llm/claude-3-5-sonnet.yaml)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*