[Feature Request]: Async extraction

[Garito]

Do you need to file a feature request?

• [x] I have searched the existing feature request and this feature request is not already filed.
• [x] I believe this is a legitimate feature request, not just a question or bug.

Feature Request Description

Hi I'm tested LightRAG for some weeks now and, pretty much, is a good solution My issues is that I have to analyse pdfs with 4k pages and more so the process never ends due to timeouts I don't really know if this project has the intent to be a professional one or the concept for the paper exposed but if LightRAG has professional inspirations issues like this should be solved Logic seems to indicate that the flow whould be: Read the pdf Chunk it Send the text chunks to the database and work jobs to a queue Then an army of workers should take every text chunk and send them for vectorization Send the vectors to the database Etc In an async fashion

Has this project professional inspirations? Thanks

Additional Context

No response

[danielaskdd]

For very large files, we recommend splitting them into smaller ones before uploading to LightRAG. This is because LightRAG processes data at the file level, and any LLM timeout or error during processing will result in the entire file failing. While retry mechanisms can leverage LLM caching to expedite previously processed content, handling extremely large files increases the complexity and risk of retry failures.

LightRAG Server supports concurrent document processing. You can control the concurrency behavior using the following environment variables:

### Maximum concurrent LLM requests (for both query and document processing)
MAX_ASYNC=6

### Number of parallel documents to process (recommended range: 2–10; set to MAX_ASYNC / 3 for optimal performance)
MAX_PARALLEL_INSERT=3

### Maximum concurrent embedding requests
EMBEDDING_FUNC_MAX_ASYNC=8

### Number of chunks sent to embedding in a single request
EMBEDDING_BATCH_NUM=16

For detailed guidance, refer to: https://github.com/HKUDS/LightRAG/blob/main/docs/LightRAG_concurrent_explain.md