when we mounting the /chroma/chroma folder on azure file share my chroma db container starting. but chromadb not working

[Anantaghatol]

What happened?

when we mounting the /chroma/chroma folder on azure file share my chroma db container starting. but chromadb not working. Please help with this. We tried tm mount chroma db on azure will it work. Thanks and regards Ananta Ghatol

Versions

Used chroma image from docker.io

Relevant log output

Starting 'uvicorn chromadb.app:app' with args: --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers --log-config chromadb/log_config.yml --timeout-keep-alive 30
DEBUG:    [22-03-2024 08:20:40] Registering provider: token_config
DEBUG:    [22-03-2024 08:20:40] Registering provider: user_token_config
DEBUG:    [22-03-2024 08:20:40] Registering provider: token
DEBUG:    [22-03-2024 08:20:40] Registering provider: token
WARNING:  [22-03-2024 08:20:41] chroma_server_nofile is set to 65535, but this is less than current soft limit of 1048576. chroma_server_nofile will not be set.
INFO:     [22-03-2024 08:20:41] Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.
DEBUG:    [22-03-2024 08:20:41] Starting component System
DEBUG:    [22-03-2024 08:20:41] Starting component OpenTelemetryClient
DEBUG:    [22-03-2024 08:20:41] Starting component SimpleAssignmentPolicy
DEBUG:    [22-03-2024 08:20:41] Starting component SqliteDB

[Anantaghatol]

@beggers please help

[beggers]

Hey @Anantaghatol,

Could you tell me a little more about the problem? It sounds like:

• You're running chromadb on Azure.
• You're using Azure Files (https://azure.microsoft.com/en-us/products/storage/files) to store your database.
• Chromadb is starting successfully, but
• It's not working.

Is this correct? If so I have questions:

• How are you running your chromadb container? In a VM, a managed offering, something else?
• Which version of chroma are you running?
• If you access the container directly, can you access the relevant file locations for storage?

My guess is that this is a permissions issue with your Azure setup and not an issue with Chroma itself, but I'm happy to help you get it working.

[Anantaghatol]

I am running Chromadb in kubernetes. I have mount path in deployment /chroma/ananta/ this path. created the files share name chroma and mounted the above directory to the file share. I am using Chromadb version 0.4.24

[tazarov]

@Anantaghatol,

If your Chroma has successfully started you should see the following in the logs:

INFO:     [29-03-2024 12:49:28] Started server process [1]
INFO:     [29-03-2024 12:49:28] Waiting for application startup.
INFO:     [29-03-2024 12:49:28] Application startup complete.
INFO:     [29-03-2024 12:49:28] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

if you don't see the above, it is possible that unicorns cannot bind to port or something else is preventing the start-up, which in turn probably leads to failing health checks resulting in your service not being accessible (assuming you have exposed Chroma via a service with ingress ctrl).

[Herrmlo]

I stumbled upon the same issue when trying to run chroma in an Azure container app where it would be stuck after following logs

2024-04-11T20:00:29.65151  Connecting to the container 'chroma'...
2024-04-11T20:00:29.66769  Successfully Connected to container: 'chroma' [Revision: 'chroma--3oex9qt-7775df45b6-wzwsf', Replica: 'chroma--3oex9qt']
2024-04-11T20:00:22.273219886Z Starting 'uvicorn chromadb.app:app' with args: --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers --log-config chromadb/log_config.yml --timeout-keep-alive 30
2024-04-11T20:00:23.363872361Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token_config
2024-04-11T20:00:23.365455792Z DEBUG:    [11-04-2024 20:00:23] Registering provider: user_token_config
2024-04-11T20:00:23.366995162Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token
2024-04-11T20:00:23.367869458Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token
2024-04-11T20:00:23.913844554Z WARNING:  [11-04-2024 20:00:23] chroma_server_nofile is set to 65535, but this is less than current soft limit of 1048576. chroma_server_nofile will not be set.
2024-04-11T20:00:24.048424921Z INFO:     [11-04-2024 20:00:24] Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.
2024-04-11T20:00:24.048443041Z DEBUG:    [11-04-2024 20:00:24] Starting component System
2024-04-11T20:00:24.048464259Z DEBUG:    [11-04-2024 20:00:24] Starting component OpenTelemetryClient
2024-04-11T20:00:24.048574959Z DEBUG:    [11-04-2024 20:00:24] Starting component SimpleAssignmentPolicy
2024-04-11T20:00:24.048655863Z DEBUG:    [11-04-2024 20:00:24] Starting component SqliteDB

and then fail any health probes and eventually be restarted.

I then found this: https://github.com/microsoft/azure-container-apps/issues/520#issuecomment-1340703979 and after adding nobrl,file_mode=0777,dir_mode=0777 to the mount options chroma started up successfully in my case.

[ndamulelonemakh]

I stumbled upon the same issue when trying to run chroma in an Azure container app where it would be stuck after following logs

2024-04-11T20:00:29.65151  Connecting to the container 'chroma'...
2024-04-11T20:00:29.66769  Successfully Connected to container: 'chroma' [Revision: 'chroma--3oex9qt-7775df45b6-wzwsf', Replica: 'chroma--3oex9qt']
2024-04-11T20:00:22.273219886Z Starting 'uvicorn chromadb.app:app' with args: --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers --log-config chromadb/log_config.yml --timeout-keep-alive 30
2024-04-11T20:00:23.363872361Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token_config
2024-04-11T20:00:23.365455792Z DEBUG:    [11-04-2024 20:00:23] Registering provider: user_token_config
2024-04-11T20:00:23.366995162Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token
2024-04-11T20:00:23.367869458Z DEBUG:    [11-04-2024 20:00:23] Registering provider: token
2024-04-11T20:00:23.913844554Z WARNING:  [11-04-2024 20:00:23] chroma_server_nofile is set to 65535, but this is less than current soft limit of 1048576. chroma_server_nofile will not be set.
2024-04-11T20:00:24.048424921Z INFO:     [11-04-2024 20:00:24] Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.
2024-04-11T20:00:24.048443041Z DEBUG:    [11-04-2024 20:00:24] Starting component System
2024-04-11T20:00:24.048464259Z DEBUG:    [11-04-2024 20:00:24] Starting component OpenTelemetryClient
2024-04-11T20:00:24.048574959Z DEBUG:    [11-04-2024 20:00:24] Starting component SimpleAssignmentPolicy
2024-04-11T20:00:24.048655863Z DEBUG:    [11-04-2024 20:00:24] Starting component SqliteDB

and then fail any health probes and eventually be restarted.

I then found this: microsoft/azure-container-apps#520 (comment) and after adding nobrl,file_mode=0777,dir_mode=0777 to the mount options chroma started up successfully in my case.

I can confirm that the suggested solution worked for me as well, I am now able to run chromadb on Azure Container Apps with persistent storage backed by File Share