1.3.1 Version 【Knowledge Retrieval】has slowed down significantly, from within one second in version 1.1.0 to over ten seconds now

[haimaxpp]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.3.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Enter the knowledge base of dify and select any document to conduct retrieval tests

2. /console/api/datasets/91739ed2-8924-4f0a-9f7f-adaa51dd0dfe/hit-testing The duration of each call to the interface exceeds 10 seconds

✔️ Expected Behavior

The result of Knowledge Retrieval should be returned soon。 When I downgraded the dify version to the original 1.1.0 version, the interface responded very quickly, which is normal

❌ Actual Behavior

from within one second in version 1.1.0 to over ten seconds now

[dosubot[bot]]

Based on the information available, the significant slowdown in Knowledge Retrieval in Dify version 1.3.1 (from under one second in 1.1.0 to over ten seconds) appears to be a known issue affecting self-hosted environments. There are a few key points to consider:

• The Knowledge Retrieval feature's performance can be impacted by configuration changes, such as retrieval modes, model settings, and especially metadata filtering and reranking options. These settings have become more configurable in recent versions and may affect speed depending on how they are set up.
• There is evidence that the cloud (SaaS) version of Dify contains hot-fixes and optimizations that are not yet available in the self-hosted version. This has led to both failures and performance issues in self-hosted deployments, as reported by other users as well. The maintainers have suggested that upgrading to the latest self-hosted version (if available) or waiting for the next release with these hot-fixes may resolve the issue. See the discussion here.
• Metadata filtering, which can be set to disabled, automatic, or manual, may also impact retrieval performance. If enabled or set to automatic/manual, it could introduce additional processing time, especially with large datasets or complex queries.

At this time, there is no specific workaround documented for version 1.3.1 other than checking your retrieval and metadata filtering configurations and monitoring for the next release that includes the relevant hot-fixes from the cloud version. If possible, try adjusting metadata filtering settings to "disabled" and see if that improves performance.

If you need to restore previous performance levels immediately, downgrading to version 1.1.0 (as you have done) is the only confirmed solution until the next self-hosted release with the necessary fixes is available.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Gevtolev]

The issue cannot be reproduced. Please provide more configuration information, such as: chunking settings, indexing method, and retrieval settings.

[EndlessSeeker]

i have the same problem. I would like to know how large your documents are?

[haimaxpp]

i have the same problem. I would like to know how large your documents are?

From several kilobytes to tens of kilobytes, they are all small files

[congjinruo]

i have the same problem. I would like to know how large your documents are?

table child_chunks Index child_chunk_dataset_id_idx was lost. try this CREATE INDEX child_chunks_index_node_id_idx ON child_chunks (index_node_id);

[dosubot[bot]]

Hi, @haimaxpp. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

• You reported a significant slowdown in Knowledge Retrieval after upgrading to Dify v1.3.1, with response times increasing from under 1 second to over 10 seconds.
• This slowdown is known in self-hosted setups, potentially caused by configuration changes and missing cloud hot-fixes.
• Recommendations included checking metadata filtering settings or downgrading to v1.1.0 as a temporary workaround.
• Maintainers requested more configuration details to reproduce the issue.
• Other users have confirmed similar slowdowns and suggested database indexing fixes.

Next Steps:

• Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open.
• Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!