Do you need to ask a question?

[x] I have searched the existing question and discussions and this question is not already answered.
[x] I believe this is a legitimate question, not just a bug or feature request.

Your Question

Hello, I imported several text documents and launched the server using:

lightrag-gunicorn --workers 4

The process ingested the first text file (there was an unhandled race condition preventing the parallel ingestion of two documents), it proceeded to the deduplication steps and completed this successfully.

Then, it ingested two text files in parallel, but after the deduplication steps it stopped with an error message suggesting it may have run out of memory. (I presume system RAM, 180GB allocated via WSL2, mostly unused).

After that, when I attempt to re-run the server in the same way, I get the below errors.

Note that I have since tested running the server with lightrag-server --workers 4 and this appears to be working (currently running), but it is re-running the entities+relationships extraction for the documents I had already processed and this took several hours (wasted).

Hence, this is to raise two questions:

Is it possible to figure out what causes the server to misbehave and handle the relevant exceptions in a graceful manner, so that no data is lost?
Does it make sense to get the lightRAG process to save intermediate outputs (which can be costly to produce) on the SSD before proceeding with the next step?

Please let me know if there is additional information I can provide to help resolve this.

Additional Context

(lightrag) ntsarb@myhostname:~/LightRAG$ lightrag-gunicorn --workers 4 2025-05-16 15:11:21 - pipmaster.package_manager - INFO - Targeting pip associated with Python: /usr/bin/python3 | Command base: /usr/bin/python3 -m pip

╔══════════════════════════════════════════════════════════════╗
║                  🚀 LightRAG Server v1.3.7/0170              ║
║          Fast, Lightweight RAG Server Implementation         ║
╚══════════════════════════════════════════════════════════════╝

📡 Server Configuration: ├─ Host: 0.0.0.0 ├─ Port: 9621 ├─ Workers: 4 ├─ CORS Origins: * ├─ SSL Enabled: False ├─ Ollama Emulating Model: lightrag:latest ├─ Log Level: INFO ├─ Verbose Debug: False ├─ History Turns: 3 ├─ API Key: Not Set └─ JWT Auth: Disabled

📂 Directory Configuration: ├─ Working Directory: /home/ntsarb/LightRAG/rag_storage └─ Input Directory: /home/ntsarb/LightRAG/inputs

🤖 LLM Configuration: ├─ Binding: ollama ├─ Host: http://localhost:11434 ├─ Model: llama3.3:70b-instruct-q8_0 ├─ Temperature: 0.2 ├─ Max Async for LLM: 4 ├─ Max Tokens: 32768 ├─ Timeout: None (infinite) ├─ LLM Cache Enabled: True └─ LLM Cache for Extraction Enabled: True

📊 Embedding Configuration: ├─ Binding: ollama ├─ Host: http://localhost:11434 ├─ Model: bge-m3:latest └─ Dimensions: 1024

⚙️ RAG Configuration: ├─ Summary Language: English ├─ Max Parallel Insert: 2 ├─ Max Embed Tokens: 8192 ├─ Chunk Size: 1200 ├─ Chunk Overlap Size: 100 ├─ Cosine Threshold: 0.2 ├─ Top-K: 60 ├─ Max Token Summary: 500 └─ Force LLM Summary on Merge: 6

💾 Storage Configuration: ├─ KV Storage: JsonKVStorage ├─ Vector Storage: NanoVectorDBStorage ├─ Graph Storage: NetworkXStorage └─ Document Status Storage: JsonDocStatusStorage

✨ Server starting up...

🌐 Server Access Information: ├─ WebUI (local): http://localhost:9621 ├─ Remote Access: http://:9621 ├─ API Documentation (local): http://localhost:9621/docs └─ Alternative Documentation (local): http://localhost:9621/redoc

📝 Note: Since the server is running on 0.0.0.0: - Use 'localhost' or '127.0.0.1' for local access - Use your machine's IP address for remote access - To find your IP address: • Windows: Run 'ipconfig' in terminal • Linux/Mac: Run 'ifconfig' or 'ip addr' in terminal

🚀 Starting LightRAG with Gunicorn 🔄 Worker management: Gunicorn (workers=4) 🔍 Preloading app: Enabled 📝 Note: Using Gunicorn's preload feature for shared data initialization

================================================================================ MAIN PROCESS INITIALIZATION Process ID: 783 Workers setting: 4

INFO: Process 783 Shared-Data created for Multiple Process (workers=4)

Starting Gunicorn with direct Python API... INFO: Process 783 Shared-Data already initialized (multiprocess=True) 2025-05-16 15:11:24,362 [INFO] lightrag: Loaded graph from /home/ntsarb/LightRAG/rag_storage/graph_chunk_entity_relation.graphml with 131 nodes, 124 edges 2025-05-16 15:11:24,374 [INFO] nano-vectordb: Load (131, 1024) data 2025-05-16 15:11:24,375 [INFO] nano-vectordb: Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': '/home/ntsarb/LightRAG/rag_storage/vdb_entities.json'} 131 data 2025-05-16 15:11:24,380 [INFO] nano-vectordb: Load (124, 1024) data 2025-05-16 15:11:24,380 [INFO] nano-vectordb: Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': '/home/ntsarb/LightRAG/rag_storage/vdb_relationships.json'} 124 data 2025-05-16 15:11:24,381 [INFO] nano-vectordb: Load (12, 1024) data 2025-05-16 15:11:24,381 [INFO] nano-vectordb: Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': '/home/ntsarb/LightRAG/rag_storage/vdb_chunks.json'} 12 data 2025-05-16 15:11:24,430 [INFO] gunicorn.error: Starting gunicorn 23.0.0

================================================================================ GUNICORN MASTER PROCESS: on_starting jobs for 4 worker(s) Process ID: 783

Memory usage after initialization: 180.19 MB LightRAG log file: /home/ntsarb/LightRAG/lightrag.log

Gunicorn initialization complete, forking workers...

2025-05-16 15:11:24,443 [INFO] gunicorn.error: Listening at: http://0.0.0.0:9621 (783) 2025-05-16 15:11:24,443 [INFO] gunicorn.error: Using worker: uvicorn.workers.UvicornWorker 2025-05-16 15:11:24,447 [INFO] gunicorn.error: Booting worker with pid: 843 INFO: Process 843 initialized updated flags for namespace: [full_docs] INFO: Process 843 ready to initialize storage namespace: [full_docs] INFO: Process 843 KV load full_docs with 1 records INFO: Process 843 initialized updated flags for namespace: [text_chunks] INFO: Process 843 ready to initialize storage namespace: [text_chunks] INFO: Process 843 KV load text_chunks with 12 records INFO: Process 843 initialized updated flags for namespace: [entities] INFO: Process 843 initialized updated flags for namespace: [relationships] INFO: Process 843 initialized updated flags for namespace: [chunks] INFO: Process 843 initialized updated flags for namespace: [chunk_entity_relation] INFO: Process 843 initialized updated flags for namespace: [llm_response_cache] INFO: Process 843 ready to initialize storage namespace: [llm_response_cache] INFO: Process 843 KV load llm_response_cache with 30 records 2025-05-16 15:11:24,542 [INFO] gunicorn.error: Booting worker with pid: 923 INFO: Process 843 initialized updated flags for namespace: [doc_status] INFO: Process 843 ready to initialize storage namespace: [doc_status] INFO: Process 843 doc status load doc_status with 7 records INFO: Process 923 storage namespace already initialized: [full_docs] INFO: Process 843 Pipeline namespace initialized INFO: Process 923 storage namespace already initialized: [text_chunks]