Weaviate search await client.connect() not setting _connection.grpc_stub on time

[Lucas-Aurelio-Costa]

trafficstars

Language: Python Version: weaviate-client 4.9.6

Context: When trying to run a query, we check if the weaviate async client is connected. When the client returns that it is connected, we run our queries. However, sometimes, the client.connect() returns before it is fully connected (grpc_stub is None).

More specifically:

await client.connect()  # This is not waiting until client is fully connected (grpc_stub)

After a few second, the grpc_stub is set and the code begins to work normally, but the first calls result in an error being raised because the client._connection.grpc_stub is None.

Client initialization:

return weaviate.use_async_with_weaviate_cloud(
        cluster_url=cluster_url,
        auth_credentials=api_key,
        additional_config=weaviate.classes.init.AdditionalConfig(timeout=weaviate.classes.init.Timeout(init=10)),
        headers=None,
        skip_init_checks=True,
    )

Connecting

async def connect_to_client():
    if not client.is_connected():
        await client.connect()

Where the error happened when trying to use the client:

  File \"/usr/local/lib/python3.11/site-packages/weaviate/collections/grpc/query.py\", line 803, in __call
    assert self._connection.grpc_stub is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

What we had to do to fix:

async def connect_to_client():
    if not client.is_connected() or client._connection.grpc_stub is None:
        async with lock:
            if not client.is_connected():
                await client.connect()

            while client._connection.grpc_stub is None:
                await asyncio.sleep(0.5)

So, we expect a fix on the await client.connect() to wait for the client to be fully connected

[clotodex]

any updates for this? running into the same issue

[tsmith023]

Hi both, are you able to post a MRE (minimum reproducible example) of your problem to highlight exactly what is going wrong here? The client.connect() method does not return until grpc_stub is set so this is highly unexpected that a subsequent query following the client.connect() call would throw an error like this!

[clotodex]

This is really hard since it has very unpredictable behavior (which made sense to me with the non-waiting). What might help: The first occurrence is unpredictable, however then it is repeatable - until i delete the container completely and relaunch it - what sometimes fixes it. @Lucas-Aurelio-Costa 's workaround has fully fixed it for me so far. This happened across versions, devices (mac+linux) (in docker compose) with the latest version being 1.28.2. I hope this helps :)