Knowledge base ingestion seems to saturate OpenAI API

[benjamin-mogensen]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

We are facing issues when creating knowledge bases via UI when we are using Excel spreadsheets as input. E.g. a 3.4mb sheet with about 3500 rows will take 3-4 hours after which it fails. The error is related to calling the embedding API outside Dify in OpenAI, have attached some screenshots.

If we run a script and use the knowledge API instead, parsing each row and sending for embedding it works fine - the OpenAI API does not seem to be "overloaded" that way. I am wondering if it is the way that Dify via UI handles the file as one large piece of data?

Steps:

1. create a new knowledge Upload excel with max chunk size 3000 and chunk overlap 100 Add delimiter /n and add in chunk setting. Index method :High quality Hybrid search and click Process After some chunks it fails with the error attached

✔️ Expected Behavior

That the whole document would be processed successfully.

❌ Actual Behavior

Embedding fails done from UI (but we are able to do row by row via knowledge API)

[dosubot[bot]]

Hello @benjamin-mogensen! I'm here to help you with bugs, questions, and becoming a contributor. Let's tackle this issue together!

I found a similar discussion regarding rate limit errors when using the Dify UI for knowledge base ingestion. The issue is that the system processes files in parallel when the UPLOAD_FILE_BATCH_LIMIT is increased, which can overwhelm the embedding service and lead to rate limit errors ^[1].

In your case, using the knowledge API to process rows individually works because it allows for sequential uploads, avoiding rate limit errors by staying within the allowed request frequency ^[1].

To address this, you might consider adjusting the UPLOAD_FILE_BATCH_LIMIT to a lower value to reduce the parallel processing load. The default value is 5, and you can modify it in the configuration file where this environment variable is defined ^[2]. Additionally, you may want to check if your subscription level allows for higher rate limits or consider contacting the service provider to increase the rate limit if needed ^[3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Yawen-1010]

Dear @benjamin-mogensen,

Your error message is a typical response when the rate limit for your current Azure OpenAI service tier is exceeded. Specifically, the error:

Error Code 429 Message: "Requests to the Embeddings_Create Operation under the Azure OpenAI API version 2024-10-01-preview have exceeded the call rate limit of your current OpenAI SO pricing tier. Please retry after 1 second. Please go here: https://aka.ms/oai/quotaincrease if you would like to further increase the default rate limit." indicates that:

Rate Limit Exceeded: The number of requests made to the Embeddings_Create operation has gone beyond what your pricing tier is set to handle. Temporary Retry: The message suggests a retry after a short pause (1 second), which is a common approach for handling transient rate limit issues. Tier Upgrade Option: There is an option to request a higher rate limit by visiting this link.

For more details on quotas and limits for Azure OpenAI services, Microsoft’s documentation provides insights into how these limits are set and what options are available for increasing them. https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits?tabs=REST#quotas-and-limits-reference

I hope this helps you resolve the issue. Since the error is related to service-side rate limits and there’s nothing further we can do from our side, I will go ahead and close this issue for now. Please feel free to reopen it or reach out if you have any other questions.