[Question]:ERROR - limit_async: Error in decorated function: Server disconnected without sending a response

[USeebi]

Do you need to ask a question?

• [x] I have searched the existing question and discussions and this question is not already answered.
• [x] I believe this is a legitimate question, not just a bug or feature request.

Your Question

The errors below generated with the Ollama server hosted on Colab and connected to a local LightRAG api server via ngrok public http tunnel on testing lightrag_ollama_demo.py.

I did search the archived issue https://github.com/HKUDS/LightRAG/issues/1640 and definitely this is not a timeout issue from disconnection initiated by the Ollama server.

I did try to isolate the problem according to suggestions from Google ai search

Network Issues:

My ngrok logs show no sign of network instability and the error disconnected every time almost at the same specific spot of embedding.

Server Issues:

I'm using a free tire of Colab and connected to a T4 GPU session. Does anyone has successful experience with LightRAG server connected to LLM server on Colab? What level of Colab resources are required? What error messages from Ollama sever to look for if the disconnection is originated from heavy loading and resource limitations?

Client Issues:

Is this possibly related to concurrency limits? How could I config for simpler or smaller embedding requests?

Any inputs beside these issues are welcome!

2025-08-24 21:24:57,894 - lightrag - INFO - Created new empty graph
2025-08-24 21:24:57,896 - lightrag - WARNING - Rerank is enabled but no rerank_model_func provided. Reranking will be skipped.
2025-08-24 21:24:57,896 - lightrag - INFO - Process 15512 KV load _full_docs with 0 records
2025-08-24 21:24:57,896 - lightrag - INFO - Process 15512 KV load _text_chunks with 0 records
2025-08-24 21:24:57,896 - lightrag - INFO - Process 15512 KV load _full_entities with 0 records
2025-08-24 21:24:57,896 - lightrag - INFO - Process 15512 KV load _full_relations with 0 records
2025-08-24 21:24:57,900 - lightrag - INFO - Process 15512 KV load _llm_response_cache with 93 records
2025-08-24 21:24:57,900 - lightrag - INFO - Process 15512 doc status load _doc_status with 0 records
2025-08-24 21:24:57,900 - lightrag - INFO - limit_async: 8 new workers initialized
2025-08-24 21:25:00,248 - lightrag - INFO - Stored 1 new unique documents
2025-08-24 21:25:00,248 - lightrag - INFO - Processing 1 document(s)
2025-08-24 21:25:00,248 - lightrag - INFO - Extracting stage 1/1: unknown_source
2025-08-24 21:25:00,248 - lightrag - INFO - Processing d-id: doc-addb4618e1697da0445ec72a648e1f92
2025-08-24 21:25:04,547 - lightrag - INFO - Chunk 1 of 42 extracted 7 Ent + 0 Rel
2025-08-24 21:25:04,547 - lightrag - INFO - Chunk 2 of 42 extracted 12 Ent + 6 Rel
2025-08-24 21:25:04,547 - lightrag - INFO - Chunk 3 of 42 extracted 5 Ent + 4 Rel
2025-08-24 21:25:04,563 - lightrag - INFO - Chunk 4 of 42 extracted 7 Ent + 4 Rel
2025-08-24 21:25:04,563 - lightrag - INFO - Chunk 5 of 42 extracted 7 Ent + 4 Rel
2025-08-24 21:25:04,563 - lightrag - INFO - Chunk 6 of 42 extracted 6 Ent + 4 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 7 of 42 extracted 5 Ent + 2 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 8 of 42 extracted 6 Ent + 3 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 9 of 42 extracted 5 Ent + 0 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 10 of 42 extracted 5 Ent + 1 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 11 of 42 extracted 4 Ent + 4 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 12 of 42 extracted 7 Ent + 0 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 13 of 42 extracted 10 Ent + 4 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 14 of 42 extracted 7 Ent + 0 Rel
2025-08-24 21:25:04,566 - lightrag - INFO - Chunk 15 of 42 extracted 6 Ent + 4 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 16 of 42 extracted 7 Ent + 4 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 17 of 42 extracted 7 Ent + 3 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 18 of 42 extracted 8 Ent + 0 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 19 of 42 extracted 8 Ent + 5 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 20 of 42 extracted 5 Ent + 3 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 21 of 42 extracted 13 Ent + 8 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 22 of 42 extracted 10 Ent + 6 Rel
2025-08-24 21:25:04,579 - lightrag - INFO - Chunk 23 of 42 extracted 7 Ent + 0 Rel
2025-08-24 21:25:04,589 - lightrag - INFO - Chunk 24 of 42 extracted 8 Ent + 2 Rel
2025-08-24 21:25:04,589 - lightrag - INFO - Chunk 25 of 42 extracted 23 Ent + 5 Rel
2025-08-24 21:25:04,589 - lightrag - INFO - Chunk 26 of 42 extracted 6 Ent + 4 Rel
2025-08-24 21:25:04,589 - lightrag - INFO - Chunk 27 of 42 extracted 7 Ent + 5 Rel
2025-08-24 21:25:04,589 - lightrag - INFO - Chunk 28 of 42 extracted 6 Ent + 3 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 29 of 42 extracted 6 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 30 of 42 extracted 15 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 31 of 42 extracted 12 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 32 of 42 extracted 13 Ent + 7 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 33 of 42 extracted 5 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 34 of 42 extracted 10 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 35 of 42 extracted 6 Ent + 1 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 36 of 42 extracted 4 Ent + 3 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 37 of 42 extracted 12 Ent + 6 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 38 of 42 extracted 7 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 39 of 42 extracted 5 Ent + 1 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 40 of 42 extracted 3 Ent + 0 Rel
2025-08-24 21:25:04,595 - lightrag - INFO - Chunk 41 of 42 extracted 6 Ent + 0 Rel
2025-08-24 21:25:04,611 - lightrag - INFO - Chunk 42 of 42 extracted 11 Ent + 0 Rel
2025-08-24 21:25:04,613 - lightrag - INFO - Merging stage 1/1: unknown_source
2025-08-24 21:25:04,614 - lightrag - INFO - Phase 1: Processing 235 entities (async: 8)
2025-08-24 21:25:04,614 - lightrag - INFO - Merge N: Charles Dickens | 3+0
2025-08-24 21:25:04,614 - lightrag - INFO - LLM merge N: Scrooge | 23+0
2025-08-24 21:25:07,457 - lightrag - INFO - LLM merge N: Marley's Ghost | 6+0
2025-08-24 21:25:07,739 - lightrag - INFO - Merge N: Marley | 2+0
2025-08-24 21:25:10,377 - lightrag - INFO - Merge N: Christmas Eve | 2+0
2025-08-24 21:25:10,378 - lightrag - INFO - Merge N: Scrooge's Nephew | 3+0
2025-08-24 21:25:11,545 - lightrag - INFO - Merge N: Clerk | 2+0
2025-08-24 21:25:12,508 - lightrag - INFO - LLM merge N: Christmas | 10+0
2025-08-24 21:25:14,119 - lightrag - INFO - LLM merge N: London | 5+0
2025-08-24 21:25:15,649 - lightrag - INFO - LLM merge N: Bob Cratchit | 6+0
2025-08-24 21:25:15,649 - lightrag - INFO - Merge N: Camden Town | 2+0
2025-08-24 21:25:15,649 - lightrag - INFO - Merge N: Jacob Marley | 2+0
2025-08-24 21:25:18,558 - lightrag - INFO - LLM merge N: Ebenezer Scrooge | 9+0
2025-08-24 21:25:22,156 - lightrag - INFO - Merge N: A Christmas Carol | 2+0
2025-08-24 21:25:27,971 - lightrag - ERROR - Error in ollama_embed: Server disconnected without sending a response.
2025-08-24 21:25:27,971 - lightrag - ERROR - limit_async: Error in decorated function: Server disconnected without sending a response.
2025-08-24 21:25:27,987 - lightrag - ERROR - Traceback (most recent call last):
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_transports\default.py", line 101, in map_httpcore_exceptions
    yield
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_transports\default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\http11.py", line 136, in handle_async_request
    raise exc
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\http11.py", line 106, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\http11.py", line 177, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpcore\_async\http11.py", line 231, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\lightrag.py", line 1396, in process_document
    await merge_nodes_and_edges(
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\operate.py", line 1309, in merge_nodes_and_edges
    raise task.exception()
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\operate.py", line 1283, in _locked_process_entity_name
    await entity_vdb.upsert(data_for_vdb)
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\kg\nano_vector_db_impl.py", line 115, in upsert
    embeddings_list = await asyncio.gather(*embedding_tasks)
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\utils.py", line 581, in wait_func
    return await future
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\utils.py", line 365, in worker
    result = await func(*args, **kwargs)
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\utils.py", line 248, in __call__
    return await self.func(*args, **kwargs)
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\llm\ollama.py", line 167, in ollama_embed
    raise e
  File "K:\Miniconda3\envs\rag\lib\site-packages\lightrag\llm\ollama.py", line 154, in ollama_embed
    data = await ollama_client.embed(
  File "K:\Miniconda3\envs\rag\lib\site-packages\ollama\_client.py", line 879, in embed
    return await self._request(
  File "K:\Miniconda3\envs\rag\lib\site-packages\ollama\_client.py", line 692, in _request
    return cls(**(await self._request_raw(*args, **kwargs)).json())
  File "K:\Miniconda3\envs\rag\lib\site-packages\ollama\_client.py", line 632, in _request_raw
    r = await self._client.request(*args, **kwargs)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_client.py", line 1629, in send
    response = await self._send_handling_auth(
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_transports\default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
  File "K:\Miniconda3\envs\rag\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "K:\Miniconda3\envs\rag\lib\site-packages\httpx\_transports\default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

Additional Context

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.