Connection hang up with gunicorn + uvicorn workers

[JanHoellmer]

When you connect to a Weaviate instance in the Gunicorn on_startup server hook and close the connection, afterwards I'm unable to connect again inside the uvicorn workers (deployment setup for FastAPI application).

gunicorn.conf.py

import weaviate
def on_starting(server):
	client = weaviate.connect_to_custom(
            http_host=localhost,
            http_port=8080,
            http_secure=False,
            grpc_host=localhost,
            grpc_port=50051,
            grpc_secure=False,
        )
	client.close()

app.py (FastAPI)

from fastapi import FastAPI
from contextlib import asynccontextmanager
import weaviate

@asynccontextmanager
async def setup_databases(app: FastAPI):
    client = weaviate.connect_to_custom(
            http_host=localhost,
            http_port=8080,
            http_secure=False,
            grpc_host=localhost,
            grpc_port=50051,
            grpc_secure=False,
        )

    yield

    weaviate_client.close()

app = FastAPI(lifespan=setup_databases)

Run with gunicorn -b 0.0.0.0:8000 -c gunicorn.conf.py app:app.

Inside the on_starting hook, i can close and connect as often as I want, same for inside the FastAPI application. As soon as i close the connection to Weaviate once in the on_starting hook, connect_to_custom hangs up inside the application. This first happened with Weaviate python client version 4.7.0 (4.7.1 & 4.8.0 also don't work), for versions <= 4.6.7 everything works fine.

Weaviate Server Version: 1.26.1

[dirkkul]

that is probably because we migrated the client to use async internally - could you try using our async client?

Here are our docs and a fastAPI example https://weaviate.io/developers/weaviate/client-libraries/python/async https://github.com/weaviate/weaviate-python-client/blob/main/journey_tests/test_fastapi.py

[tsmith023]

Hi @JanHoellmer, I'll look into exactly what is causing the hang with gunicorn but be advised that this deployment method for FastAPI is deprecated and the current recommendation is to deploy using just uvicorn

[JanHoellmer]

Hi @JanHoellmer, I'll look into exactly what is causing the hang with gunicorn but be advised that this deployment method for FastAPI is deprecated and the current recommendation is to deploy using just uvicorn

Thanks for the hint, have not seen that yet.

[tsmith023]

If you choose to move away from gunicorn, it would be interesting to know whether this fixes your problem here. Likewise, if you refactor to use the WeaviateAsyncClient object it would be great to know if that resolves the issue!

[JanHoellmer]

I tested both:

• Moving away from gunicorn to just uvicorn helps, but probably because there is no direct substitute for the on_startup server hook from gunicorn.
• Using WeaviateAsyncClient, while still using gunicorn, also fixes the hang up. In on_startup you can still use the sync WeaviateClient, but as soon as you only use WeaviateAsyncClient on the FastAPI side, the connection can be astablished normally. (I tested the other way around, using WeaviateAsyncClient in gunicorn, and WeaviateClient in FastAPI -> also works!) So the problem only persists when using WeaviateClient on both ends.

But thanks again for the hint with uvicorn, I'm gonna migrate, but maybe the bug is still relevant for other setups.

[tsmith023]

@JanHoellmer, when you say

on_startup server hook from gunicorn

do you mean this one?

If so, this is entirely inline with the client implementation as it is not multi-processing safe so if it is initialised in one process and then used in another then it won't function correctly as you observed!

If not, which other hook were you using specifically that was causing the hanging behaviour?

[JanHoellmer]

Yes, it is the one you linked. But the async client is multi-processing safe? Or why does it not happen there, that when i initialize and close the connection in one proces (in the on_startup hook), i can't initialize it again in another process?

[tsmith023]

Hi @JanHoellmer, sorry for letting this one go stale! In general, the client is not safe to be shared across multiple processes. If you are in such an environment then you should create a single client within each process. This is because there is some statefulness in the client wrt to connection pooling in the httpx and grpc layers

[tsmith023]

Hi @JanHoellmer, we recently refactored the underlying connection for the sync/async clients and I think this issue may have been fixed by the refactor. Can you try using v4.13.0 and confirm whether you're still seeing this hangup issue? Cheers!

[dirkkul]

Closing this, could you please reopen in case there are still issues?