[Install issue]: Trying to deploy chromadb as a Google Cloud Run Service

[solvek]

What happened?

Created a service. Deployed directly docker from Docker Hub.

Now trying connect via HttpClient:

chroma_client = chromadb.HttpClient(host="<MY_HOST>",
    port='443',
    ssl=True,
    settings=Settings(allow_reset=True, anonymized_telemetry=False),
    headers={'authorization': 'PASS'})

Versions

PYTHON_VERSION=3.10.13 chromadb 0.4.18

Relevant log output

Client size

HTTPError                                 Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/chromadb/api/fastapi.py in raise_chroma_error(resp)
    627     try:
--> 628         resp.raise_for_status()
    629     except requests.HTTPError:

9 frames
HTTPError: 502 Server Error: Bad Gateway for url: https://<MY_HOST>:443/api/v1/tenants/default_tenant

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
Exception: upstream connect error or disconnect/reset before headers. reset reason: protocol error

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/chromadb/api/client.py in _validate_tenant_database(self, tenant, database)
    431             )
    432         except Exception:
--> 433             raise ValueError(
    434                 f"Could not connect to tenant {tenant}. Are you sure it exists?"
    435             )

ValueError: Could not connect to tenant default_tenant. Are you sure it exists?

Server side

NOTICE 2023-11-27T16:24:11.903725Z [protoPayload.serviceName: run.googleapis.com] [protoPayload.methodName: google.cloud.run.v1.Services.ReplaceService] [protoPayload.resourceName: namespaces/harmonia-9dead/services/chroma] [protoPayload.authenticationInfo.principalEmail: ] audit_log, method: "google.cloud.run.v1.Services.ReplaceService", principal_email: ""
DEFAULT 2023-11-27T16:24:19.557723Z Rebuilding hnsw to ensure architecture compatibility
DEFAULT 2023-11-27T16:24:23.548897Z Collecting chroma-hnswlib
DEFAULT 2023-11-27T16:24:23.632004Z Downloading chroma_hnswlib-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
DEFAULT 2023-11-27T16:24:23.900415Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 8.9 MB/s eta 0:00:00
DEFAULT 2023-11-27T16:24:24.514677Z Collecting numpy
DEFAULT 2023-11-27T16:24:24.531744Z Downloading numpy-1.26.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
DEFAULT 2023-11-27T16:24:24.984832Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 61.9 MB/s eta 0:00:00
DEFAULT 2023-11-27T16:24:25.807177Z Installing collected packages: numpy, chroma-hnswlib
DEFAULT 2023-11-27T16:24:25.807428Z Attempting uninstall: numpy
DEFAULT 2023-11-27T16:24:25.810576Z Found existing installation: numpy 1.26.2
DEFAULT 2023-11-27T16:24:26.411828Z Uninstalling numpy-1.26.2:
DEFAULT 2023-11-27T16:24:32.542974Z Successfully uninstalled numpy-1.26.2
DEFAULT 2023-11-27T16:24:35.263417Z Attempting uninstall: chroma-hnswlib
DEFAULT 2023-11-27T16:24:35.265361Z Found existing installation: chroma-hnswlib 0.7.3
DEFAULT 2023-11-27T16:24:35.272839Z Uninstalling chroma-hnswlib-0.7.3:
DEFAULT 2023-11-27T16:24:35.769899Z Successfully uninstalled chroma-hnswlib-0.7.3
DEFAULT 2023-11-27T16:24:35.846848Z Successfully installed chroma-hnswlib-0.7.3 numpy-1.26.2
DEFAULT 2023-11-27T16:24:35.847410Z WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
DEFAULT 2023-11-27T16:24:36.034065Z [notice] A new release of pip is available: 23.0.1 -> 23.3.1
DEFAULT 2023-11-27T16:24:36.034077Z [notice] To update, run: pip install --upgrade pip
DEFAULT 2023-11-27T16:24:41.550696Z WARNING: [27-11-2023 16:24:41] chroma_server_nofile cannot be set to a value greater than the current hard limit of 25000. Keeping soft limit at 25000
DEFAULT 2023-11-27T16:24:42.047506Z INFO: [27-11-2023 16:24:42] Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
DEFAULT 2023-11-27T16:24:42.047753Z DEBUG: [27-11-2023 16:24:42] Starting component System
DEFAULT 2023-11-27T16:24:42.047917Z DEBUG: [27-11-2023 16:24:42] Starting component OpenTelemetryClient
DEFAULT 2023-11-27T16:24:42.047984Z DEBUG: [27-11-2023 16:24:42] Starting component SimpleAssignmentPolicy
DEFAULT 2023-11-27T16:24:42.048086Z DEBUG: [27-11-2023 16:24:42] Starting component SqliteDB
DEFAULT 2023-11-27T16:24:42.100919Z DEBUG: [27-11-2023 16:24:42] Starting component Posthog
DEFAULT 2023-11-27T16:24:42.101093Z DEBUG: [27-11-2023 16:24:42] Starting component LocalSegmentManager
DEFAULT 2023-11-27T16:24:42.101199Z DEBUG: [27-11-2023 16:24:42] Starting component SegmentAPI
DEFAULT 2023-11-27T16:24:42.217160Z INFO: [27-11-2023 16:24:42] Started server process [7]
DEFAULT 2023-11-27T16:24:42.217244Z INFO: [27-11-2023 16:24:42] Waiting for application startup.
DEFAULT 2023-11-27T16:24:42.218141Z INFO: [27-11-2023 16:24:42] Application startup complete.
DEFAULT 2023-11-27T16:24:42.220735Z INFO: [27-11-2023 16:24:42] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO 2023-11-27T16:24:42.239789Z Default STARTUP TCP probe succeeded after 1 attempt for container "chroma-1" on port 8000.
INFO 2023-11-27T16:24:42.332790Z [protoPayload.serviceName: run.googleapis.com] [protoPayload.methodName: v1] [protoPayload.resourceName: namespaces/harmonia-9dead/revisions/chroma-00010-c9l] Ready condition status changed to True for Revision chroma-00010-c9l.
INFO 2023-11-27T16:24:47.688381Z [protoPayload.serviceName: run.googleapis.com] [protoPayload.methodName: v1] [protoPayload.resourceName: namespaces/harmonia-9dead/services/chroma] Ready condition status changed to True for Service chroma.
ERROR 2023-11-27T16:25:14.365892Z [httpRequest.requestMethod: GET] [httpRequest.status: 502] [httpRequest.responseSize: 969 B] [httpRequest.latency: 2 ms] [httpRequest.userAgent: python-requests 2.31.0] https://<MY_HOST>/api/v1/tenants/default_tenant
DEFAULT 2023-11-27T16:25:14.378587Z WARNING: [27-11-2023 16:25:14] Invalid HTTP request received.

[PhlipPhlops]

With the info I see here, my first assumption is that this port configuration issue.

• I see you're connecting to your chromadb.HttpClient to port='443', I assume on your Cloud Run container URL.
• And I see the chroma server is running within the container on port 8000: Uvicorn running on http://0.0.0.0:8000

According to this cloud run container article, cloud run forwards incoming requests to port 8080 within the container, which doesn't match the port 8000 that chroma listens on

Niftily-enough you might be able to fix this with a single gcloud command. gcloud run services update SERVICE --port PORT, where SERVICE is the name of your running service and PORT is what you want gcloud to forward requests to (Chroma's 8000)

I'm curious if that does the trick, please let me know!

[solvek]

Tried this command

gcloud run services update SERVICE --port PORT, where SERVICE is the name of your running service and PORT is what you want gcloud to forward requests to (Chroma's 8000)

But this didn't help. The port 8000 was already configured.

According to my logs from server it seems the requests are forwarded fine to port 8000 and are handled by server... howerver the path /api/v1/tenants/default_tenant seems to be not working :(

[jeffchuber]

@tazarov any ideas here?

[solvek]

It seems I managed to sort this out. I am going to document how to deploy chroma server as a Google Cloud Run Service and how to connect to it from Google Cloud Run Job. However before this want to resolve some other issues I have. The first issue is I cannot reset the db even thougt I have set allow_reset=True

Creating client

resetting_client = chromadb.HttpClient(
    host=CHROMA_SERVER_HOST,
    port=CHROMA_SERVER_HTTP_PORT,
    ssl = True,
    settings=Settings(anonymized_telemetry=False, allow_reset=True)
    )

Then

resetting_client.reset()

However getting error Exception: {"error":"ValueError('Resetting is not allowed by this configuration (to enable it, set allow_resettoTruein your Settings() or includeALLOW_RESET=TRUE in your environment variables)')"}

[tazarov]

@solvek, reset is a flag you need to set on the server-side e.g. your Cloud Run Service cofig needs to contain ALLOW_RESET=TRUE env var.

[solvek]

ALLOW_RESET=TRUE on server side helped me. Thanks.

Another strange issue is I am getting in logs warnings about deprecated configuration (and telemetry btw):

2023-11-28 17:47:46.927 EET
Failed to send telemetry event ClientStartEvent: [91mYou are using a deprecated configuration of Chroma.
2023-11-28 17:47:46.927 EET
Connected to chromadb
2023-11-28 17:47:46.927 EET
[94mIf you do not have data you wish to migrate, you only need to change how you construct
2023-11-28 17:47:46.927 EET
your Chroma client. Please see the "New Clients" section of https://docs.trychroma.com/migration.
2023-11-28 17:47:46.927 EET
________________________________________________________________________________________________
2023-11-28 17:47:46.927 EET
If you do have data you wish to migrate, we have a migration tool you can use in order to
2023-11-28 17:47:46.927 EET
migrate your data to the new Chroma architecture.
2023-11-28 17:47:46.927 EET
Please `pip install chroma-migrate` and run `chroma-migrate` to migrate your data and then
2023-11-28 17:47:46.927 EET
change how you construct your Chroma client.
2023-11-28 17:47:46.927 EET
See https://docs.trychroma.com/migration for more information or join our discord at https://discord.gg/8g5FESbj for help![0m

This is my code on the client side:

    print(f"Starting Task #{TASK_INDEX}, Attempt #{TASK_ATTEMPT}...")
    
    client = chromadb.HttpClient(
        host=CHROMA_SERVER_HOST,
        port=CHROMA_SERVER_HTTP_PORT,
        ssl = True,
        settings=Settings(anonymized_telemetry=False)
        )
    print("Connected to chromadb")

    default_ef = embedding_functions.DefaultEmbeddingFunction()

    collection = client.get_or_create_collection(
        "all-my-documents",
        embedding_function=default_ef)

    collection.add(
        documents=["This is document1", "This is document2"],
        metadatas=[{"source": "notion"}, {"source": "google-docs"}], # filter on these!
        ids=["doc1", "doc2"], # unique for each doc
    )
    print("Chroma docs added")

[solvek]

And the 3rd issue I have an error:

Task #0, Attempt #0 failed: You must provide an embedding function to compute embeddings.https://docs.trychroma.com/embeddings

However I am providing embedding function as you can see in my code snippet above.

[tazarov]

@solvek, what Chroma version is you client?

EDIT: The client code looks fine. However, you should note that embedding_functions.DefaultEmbeddingFunction() will trigger the download of onyx runtime + all-MiniLM model (83MB file). If you run this in a serverless, it might only be optimal if there is some caching. As an alternative, I suggest you switch to either OpenAI, Cohere or HF inference API.

[solvek]

Chroma client version is 0.4.18 Even thoght I am setting embedding function I am getting error when I add data to my collection. (On production I will use Palm2 however for now I want to have any simple working solution without errors)

Task #0, Attempt #0 failed: You must provide an embedding function to compute embeddings.https://docs.trychroma.com/embeddings

P.S. Probably some dependencies are missing. I am using chromadb-client package on the Cloud Job side.

[solvek]

Additional information: It seems I am having all these issues because I am user chromadb-client package instead of chromadb. chromadb does not work on the cloud job at all. Described this in another issue here: https://github.com/GoogleCloudPlatform/buildpacks/issues/366

[tazarov]

@solvek, chroma-client is the client-only version of Chroma, and it has very little dependencies:

dependencies = [
  'requests >= 2.28',
  'pydantic>=1.9',
  'numpy >= 1.22.5',
  'posthog >= 2.4.0',
  'typing_extensions >= 4.5.0',
  'overrides >= 7.3.1',
]

As far as I know GCR operates on top of Knative functions. Knative functions usually require you to build an image. Do you have the Dockerfile of your image?

[tazarov]

Nvm I found it from the linked issue:

  "telethon",
  "langchain >= 0.0.338",
  "google-cloud-aiplatform",
  "chromadb ==0.4.18"
]

Can you change chromadb==0.4.18 to chromadb-client==0.4.18.dev0

[solvek]

I do not have a Dockerfile. I am building the container from python code. This is my requirements.txt:

opentelemetry-sdk
opentelemetry-exporter-otlp
chromadb-client

I tried also to run the client in Google Colab and really if I install package chromadb then I do not have the issue with embeddings, when I install only chromadb-client I am having this error there as well. So it seems some dependency is missing.

[solvek]

With chromadb-client==0.4.18.dev0 still getting that error with embeddings

[solvek]

By the way when using chromadb-client I had to add also

opentelemetry-sdk
opentelemetry-exporter-otlp

Otherwise it would not work even if I have anonymized_telemetry=False

[tazarov]

Alright then you might be onto something. Let me do some digging on my side for a moment.

[tazarov]

@solvek, Looking at the code, this is intended behaviour:

def DefaultEmbeddingFunction() -> Optional[EmbeddingFunction[Documents]]:
    if is_thin_client:
        return None
    else:
        return ONNXMiniLM_L6_V2()

is_thin_client is set for the chromadb-client package. The issue you're facing is that to run the ONNX runtime and all, you need a few more dependencies, so we've opted not to include those for the lightweight client.

As a workaround, I would suggest trying the following:

• add onnxruntime>=1.14.1 to your requirements
• Instead of ef = embedding_functions.DefaultEmbeddingFunction() try ef = embedding_functions.ONNXMiniLM_L6_V2() (this still might fail due to some deps)

[tazarov]

@solvek, to summarize our discussion in Discord.

• The above workaround revealed a problem in our chromadb-client package where missing OTEL deps were causing failure
• The Google Cloud Run build pack needed a downgrade to Python 3.11 (it was 3.12)
• In addition to onnxruntime, tokenizers package needed install
• Finally, setting ef.DOWNLOAD_PATH = "models/ONNXMiniLM_L6_V2" needed to be added to resolve the permissions error

The bottom line is while the above worked, it is not a workable solution as GCR is a serverless function for which downloading 80MB on each run is a non-starter. The appropriate solution is to use an external API EF (OpenAI, Cohere, Vertex, HF etc).

I think with the above learnings and the documentation about how to set and run Chroma thin client in a GCR (for which I'll create a separate issue), we can safely close this issue.

[florind]

Wouldn't you want to mount a persistent volume to this as well? I've only found an in-memory volume mount option for Cloud Run and the data won't survive. Am I missing something?

[tazarov]

@florind, granted you could store the data in the GCR persistent volume and check if the model exists in it, if yes then load and use that.

Generally speaking doing this in a serverless fashion is not optimal as loading and running models is not the fastest thing. I would much rather use a separate service either an API or another Container service in Google with HF's embedding server. We recently added support and instructions how to run it - https://docs.trychroma.com/embeddings/hugging-face-embedding-server (in docker)

[debkanchan]

It seems I managed to sort this out. I am going to document how to deploy chroma server as a Google Cloud Run Service and how to connect to it from Google Cloud Run Job. However before this want to resolve some other issues I have. The first issue is I cannot reset the db even thougt I have set allow_reset=True

Creating client

resetting_client = chromadb.HttpClient(
    host=CHROMA_SERVER_HOST,
    port=CHROMA_SERVER_HTTP_PORT,
    ssl = True,
    settings=Settings(anonymized_telemetry=False, allow_reset=True)
    )

Then

resetting_client.reset()

However getting error Exception: {"error":"ValueError('Resetting is not allowed by this configuration (to enable it, set allow_resettoTruein your Settings() or includeALLOW_RESET=TRUE in your environment variables)')"}

I have run into the same issue. Could you please document how you sorted the connection out?

[debkanchan]

Update: You have to mount colume to /chroma/chroma and disable http2 on cloud run