When cyclically calling the knowledge base API to retrieve segments one by one, the first few segments are always output repeatedly.

[389703328]

Self Checks

• [x] I have read the Contributing Guide and Language Policy.
• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report, otherwise it will be closed.
• [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Why do the first few segments always repeat when I cyclically call the knowledge base API to retrieve segments one by one? The page number is set to increment, and the limit is 1. I saw that someone had asked a similar repeated question last year, but it wasn't resolved？

✔️ Expected Behavior

Correct sequential segment output.

❌ Actual Behavior

Duplicate output

[dosubot[bot]]

This issue happens because the API's pagination logic relies on the DocumentSegment.position field to order segments, and uses offset/limit-based pagination via db.paginate in the backend. If the position values are not unique or stable, or if the database query does not consistently order segments, you can get repeated outputs even when incrementing the page number and setting limit=1 (source). There is no explicit bug fix or workaround for this in Dify 1.11.0, and recent code changes have not addressed this specific problem.

To avoid duplicates, check that your segments have unique and stable position values in the database. If you control the data, ensure that new segments are inserted with incrementing, unique positions. If you still see duplicates, you may need to review or patch the segment creation logic to enforce this ordering (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}