dify icon indicating copy to clipboard operation
dify copied to clipboard

New documents stuck in Indexing and Queing in the knowledge base

Open ciriatico-sol opened this issue 3 weeks ago • 9 comments

Self Checks

  • [x] I have read the Contributing Guide and Language Policy.
  • [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
  • [x] I have searched for existing issues search for existing issues, including closed ones.
  • [x] I confirm that I am using English to submit this report, otherwise it will be closed.
  • [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
  • [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.1

Cloud or Self Hosted

Cloud

Steps to reproduce

Upload a new document to the knowledge base.

✔️ Expected Behavior

It's expected that the document is processed and indexed.

❌ Actual Behavior

The document gets stuck in the indexing or queing step (it's been stuck for more than 30 minutes).

Image

ciriatico-sol avatar Dec 18 '25 14:12 ciriatico-sol

Documents stuck in "Indexing" or "Queuing" in Dify Cloud (v1.11.1) usually point to issues in the async processing pipeline. The main causes are: Redis queue or TTL problems (tasks not dequeued), Celery worker outages (no worker available), upstream provider/model errors (embedding/model failures), document paused/deleted states, or hitting subscription/batch upload limits (which set status to "error") — see the technical details in the codebase (source, source, source).

You can check for error messages or status fields in the Dify UI (look for an "error" column or details in the document row). If the document is stuck but not marked as "error", it may be waiting for a worker or queue slot. For cloud users, diagnostics are limited to the UI; backend logs are not exposed.

Recommended actions: try re-uploading the document, check for any error messages in the UI, and note the document ID and timestamps. If the document remains stuck for over an hour (the default queue TTL), collect the document ID, upload time, and any visible error details to help with further diagnosis. If you hit subscription or batch upload limits, the UI should show an error status and message.

This issue has been reported by other users, so it's not isolated (example). If you need to escalate, providing document IDs, timestamps, and any error messages will help maintainers investigate.

To reply, just mention @dosu.


How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

dosubot[bot] avatar Dec 18 '25 14:12 dosubot[bot]

I've tried uploading the document again, and the problem persists.

Dataset ID: d71363b1-fa52-4dea-8ffc-bd15e2d9a01b Document ID: 4f67fd75-6248-43b6-92f9-b03cdf7017f1 Upload Date: December 18, 2025 11:22 AM

ciriatico-sol avatar Dec 18 '25 14:12 ciriatico-sol

Since re-uploading didn't resolve the stuck indexing/queuing, here are the next steps and what you can check in Dify Cloud:

  • The UI shows document status ("Queuing", "Indexing", etc.) and may display an error tooltip if the backend provides an error message. Click into the document detail page to see if any error is surfaced—this can help diagnose the root cause.
  • You can manually pause and resume indexing from the document detail view, and trigger a "sync" operation for some document types. These actions sometimes help move a stuck document forward, but there is no explicit "force re-index" button in the UI. If the document is stuck and not in "error", these controls are your only options for intervention (UI details).
  • If you see an error tooltip, copy the error text—it often points to issues like missing vector database configuration (e.g., a missing "default" named vector in Qdrant or Weaviate), which is a known cause for stuck documents (reference).
  • For maintainers, the most useful info is: document status, any error message, Dataset ID, Document ID, upload timestamp, and a screenshot of the UI. This helps them correlate your case with backend logs and known issues.

If the document remains stuck after pause/resume/sync and no error is shown, it's likely a backend queue or resource issue. Since you’re on cloud, you can’t access logs directly, so providing the above details is the best way to help the team investigate.

To reply, just mention @dosu.


How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

dosubot[bot] avatar Dec 18 '25 14:12 dosubot[bot]

I've tried uploading the document again, and the problem persists.

Dataset ID: d71363b1-fa52-4dea-8ffc-bd15e2d9a01b Document ID: 4f67fd75-6248-43b6-92f9-b03cdf7017f1 Upload Date: December 18, 2025 11:22 AM

check the dify worker docker log, maybe there is some error

fatelei avatar Dec 18 '25 14:12 fatelei

I've tried uploading the document again, and the problem persists. Dataset ID: d71363b1-fa52-4dea-8ffc-bd15e2d9a01b Document ID: 4f67fd75-6248-43b6-92f9-b03cdf7017f1 Upload Date: December 18, 2025 11:22 AM

check the dify worker docker log, maybe there is some error

I'm using the cloud version, so I have few log info

ciriatico-sol avatar Dec 18 '25 15:12 ciriatico-sol

I've tried uploading the document again, and the problem persists. Dataset ID: d71363b1-fa52-4dea-8ffc-bd15e2d9a01b Document ID: 4f67fd75-6248-43b6-92f9-b03cdf7017f1 Upload Date: December 18, 2025 11:22 AM

check the dify worker docker log, maybe there is some error

I'm using the cloud version, so I have few log info

you can reindex the document

fatelei avatar Dec 19 '25 01:12 fatelei

I also encountered the same problem. Two weeks ago, I saw this issue #28994 . My code is now up-to-date, but the problem still exists

Devindan avatar Dec 19 '25 07:12 Devindan

the problem still exists v1.11.2

lywy233 avatar Dec 26 '25 11:12 lywy233

the problem still exists v1.11.2

you can check the dify worker log

fatelei avatar Dec 26 '25 12:12 fatelei

@fatelei @Devindan @lywy233 @ciriatico-sol @appleboy This may help you!! #30237

ChillPilot avatar Dec 27 '25 07:12 ChillPilot