Timeout in python client working but still returning timeout error

[zstatmanweil]

Versions

qdrant_client version: 11.1 qdrant version: 0.11.3 python: 3.9

Summary

I have been getting timeouts when creating collections. I passed in timeout=60 to the recreate_collection method and it still appears to timeout after about 10 seconds. See stack trace below. However, when I look at the logs in the cluster, I see it did appear to successfully pass the timeout parameter to the PUT request: [2022-11-21T20:01:05.375Z INFO actix_web::middleware::logger] 10.2.137.204 "PUT /collections/test_v01_32_16_10?timeout=60 HTTP/1.1" 200 72 "-" "python-httpx/0.23.0" 60.814669

Timeout error from python client

Traceback (most recent call last):
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/backends/sync.py", line 26, in read
    return self._sock.recv(max_bytes)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 90, in handle_request
    return self._connection.handle_request(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 105, in handle_request
    raise exc
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 84, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 148, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 177, in _receive_event
    data = self._network_stream.read(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/backends/sync.py", line 26, in read
    return self._sock.recv(max_bytes)
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api_client.py", line 95, in send_inner
    response = self._client.send(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_client.py", line 902, in send
    response = self._send_handling_auth(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_client.py", line 930, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_client.py", line 967, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_client.py", line 1003, in _send_single_request
    response = transport.handle_request(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/zoestatman-weil/code/earth-index-ml/indexing/qdrant/tiles/create_collections_for_tiles.py", line 65, in <module>
    main(host=host,
  File "/Users/zoestatman-weil/code/earth-index-ml/indexing/qdrant/tiles/create_collections_for_tiles.py", line 25, in main
    client.recreate_collection(collection_name=qdrant_collection,
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/qdrant_client.py", line 1191, in recreate_collection
    self.http.collections_api.create_collection(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api/collections_api.py", line 618, in create_collection
    return self._build_for_create_collection(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api/collections_api.py", line 193, in _build_for_create_collection
    return self.api_client.request(
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api_client.py", line 68, in request
    return self.send(request, type_)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api_client.py", line 85, in send
    response = self.middleware(request, self.send_inner)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api_client.py", line 188, in __call__
    return call_next(request)
  File "/Users/zoestatman-weil/code/earth-index-ml/venv/lib/python3.9/site-packages/qdrant_client/http/api_client.py", line 97, in send_inner
    raise ResponseHandlingException(e)
qdrant_client.http.exceptions.ResponseHandlingException: timed out

[anammari]

Same issue here.

qdrant-client==0.11.3 qdrant container version: 0.11.4 python: 3.9.15

Qdrant Docker container looks fine:

` 2022-12-06 11:25:57 Access web UI at https://ui.qdrant.tech/?v=v0.11.4

2022-12-06 11:25:57

2022-12-06 11:25:26 [2022-12-06T00:25:26.391Z INFO actix_server::worker] Shutting down idle worker

2022-12-06 11:25:26 [2022-12-06T00:25:26.391Z INFO actix_server::worker] Shutting down idle worker

2022-12-06 11:25:26 [2022-12-06T00:25:26.391Z INFO actix_server::worker] Shutting down idle worker

2022-12-06 11:25:26 [2022-12-06T00:25:26.392Z INFO actix_server::worker] Shutting down idle worker

2022-12-06 11:25:26 [2022-12-06T00:25:26.392Z INFO actix_server::worker] Shutting down idle worker

2022-12-06 11:25:57 [2022-12-06T00:25:57.249Z INFO storage::content_manager::consensus::persistent] Loading raft state from ./storage/raft_state

2022-12-06 11:25:57 [2022-12-06T00:25:57.445Z INFO storage::content_manager::toc] Loading collection: my_collection

2022-12-06 11:26:03 [2022-12-06T00:26:03.452Z INFO qdrant] Distributed mode disabled

2022-12-06 11:26:03 [2022-12-06T00:26:03.457Z INFO actix_server::builder] Starting 15 workers

2022-12-06 11:26:03 [2022-12-06T00:26:03.458Z INFO actix_server::server] Actix runtime found; starting in Actix runtime

2022-12-06 11:26:03 [2022-12-06T00:26:03.878Z INFO qdrant::tonic] Qdrant gRPC listening on 6334

2022-12-06 11:26:28 [2022-12-06T00:26:28.907Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/my_collection HTTP/1.1" 200 69 "-" "python-httpx/0.23.1" 2.247990

2022-12-06 11:26:47 [2022-12-06T00:26:47.882Z INFO actix_web::middleware::logger] 172.17.0.1 "PUT /collections/my_collection HTTP/1.1" 200 70 "-" "python-httpx/0.23.1" 18.941867

2022-12-06 11:34:09 [2022-12-06T00:34:09.338Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/my_collection HTTP/1.1" 200 69 "-" "python-httpx/0.23.1" 1.419514

2022-12-06 11:34:27 [2022-12-06T00:34:27.671Z INFO actix_web::middleware::logger] 172.17.0.1 "PUT /collections/my_collection HTTP/1.1" 200 70 "-" "python-httpx/0.23.1" 18.304708 `

The only change I did to Docker is the use of the WSL 2 based engine (Settings -> General -> tick Use the WSL 2 based engine) but I cannot confirm if this is the cause of the issue.

[calix]

I had the exact same issue. Traced it to a request sent in the httpx library. Seems like the response given is 400 Bad Request, but this is somewhere incorrectly pushed up the exception chain as a timeout.

Solved the issue by using the docker pull qdrant/qdrant:v1.1.1 docker image version instead of latest.

[cnmoro]

Same error here.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/qdrant_client/http/api_client.py", line 95, in send_inner
    response = self._client.send(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.WriteTimeout: timed out

[jonaskyhse]

It seems httpx has a default timeout for its clients https://www.python-httpx.org/advanced/#timeout-configuration. if the qdrant client doesn't alter this when requesting it will timeout. However, it seems to me that this is in fact done through the timeout parameter by passing **kwargs to httpx.Client(). I am wondering if this could be an issue with httpx?

Btw. This happens to me on the snapshot API all the time. When setting the timeout to some arbitrary high number I don't get this issue. Obviously if the snapshot process is done before the timeout.

[amn-max]

I facing the same issue on windows when creating and re-creating collection. I am using qdrant client version v1.7.0 and qdrant version v1.7.0. However it works perfectly on linux systems.

I have done some trial and error testing. I have found that

1. When I run using the docker run -p 6333:6333 qdrant/qdrant directly on terminal. Works perfectly
2. When using docker compose throws the same timeout error. Here is my yml for reference

    qdrant:
        image: qdrant/qdrant:v1.7.0
        container_name: 'vector-db'
        ports:
          - "6333:6333"
        expose:
          - "6333"
        volumes:
          - "./vbs:/qdrant/storage"

This is the error

Traceback (most recent call last):
  File "E:\ProJects\venv\lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions
    yield
  File "E:\ProJects\venv\lib\site-packages\httpcore\_backends\sync.py", line 126, in read
    return self._sock.recv(max_bytes)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\ProJects\venv\lib\site-packages\httpx\_transports\default.py", line 67, in map_httpcore_exceptions
    yield
  File "E:\ProJects\venv\lib\site-packages\httpx\_transports\default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 268, in handle_request
    raise exc
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\connection.py", line 103, in handle_request
    return self._connection.handle_request(request)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\http11.py", line 133, in handle_request
    raise exc
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\http11.py", line 111, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\http11.py", line 176, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_sync\http11.py", line 212, in _receive_event
    data = self._network_stream.read(
  File "E:\ProJects\venv\lib\site-packages\httpcore\_backends\sync.py", line 126, in read
    return self._sock.recv(max_bytes)
  File "C:\Users\ayush\.pyenv\pyenv-win\versions\3.9.7\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "E:\ProJects\venv\lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api_client.py", line 101, in send_inner
    response = self._client.send(request)
  File "E:\ProJects\venv\lib\site-packages\httpx\_client.py", line 915, in send
    response = self._send_handling_auth(
  File "E:\ProJects\venv\lib\site-packages\httpx\_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
  File "E:\ProJects\venv\lib\site-packages\httpx\_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
  File "E:\ProJects\venv\lib\site-packages\httpx\_client.py", line 1016, in _send_single_request
    response = transport.handle_request(request)
  File "E:\ProJects\venv\lib\site-packages\httpx\_transports\default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "C:\Users\ayush\.pyenv\pyenv-win\versions\3.9.7\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "E:\ProJects\venv\lib\site-packages\httpx\_transports\default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\ProJects\some.py", line 10, in <module>
    qdrant_client.recreate_collection(
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\qdrant_client.py", line 1751, in recreate_collection
    return self._client.recreate_collection(
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\qdrant_remote.py", line 2183, in recreate_collection
    return self.create_collection(
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\qdrant_remote.py", line 2154, in create_collection
    result: Optional[bool] = self.http.collections_api.create_collection(
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api\collections_api.py", line 1118, in create_collection
    return self._build_for_create_collection(
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api\collections_api.py", line 96, in _build_for_create_collection
    return self.api_client.request(
  File "E:\ProJectsvenv\lib\site-packages\qdrant_client\http\api_client.py", line 74, in request
    return self.send(request, type_)
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api_client.py", line 91, in send
    response = self.middleware(request, self.send_inner)
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api_client.py", line 200, in __call__
    return call_next(request)
  File "E:\ProJects\venv\lib\site-packages\qdrant_client\http\api_client.py", line 103, in send_inner
    raise ResponseHandlingException(e)
qdrant_client.http.exceptions.ResponseHandlingException: timed out

Please let me know if I am doing anything wrong here?

[giacomov]

I found a workaround, which is ugly but works for now. If client is your Python SDK client, then you can set the timeout for HTTPx with:

client._client.openapi_client.client._client.timeout = 60  # seconds

as I said, very ugly :-)

[joein]

should be fixed as of qdrant-client==1.8.2