[Feature Request]: Fly.io Deploy

[patrickbrophy]

Describe the problem

Deploying to Fly.io would simplify for self hosting chroma. Fly's deployments make it super simple to deploy with docker containers.

Describe the proposed solution

I am beginning work in my own personal fork of chroma. Fly.io has a terraform provider.

Currently my deploy is not yet working and eventually I want to make it more configurable. My deploy is based off of the Google Cloud deploy. I believe it is possible to deploy with Fly considering that they have docker support, I just need to figure out how to get the Docker Container to build properly on Fly. Currently the container fails to build the hnsw python package. GCC will crash due to a lack of memory. I am considering trying to scale up the amount of memory and compute to fix this.

Alternatives considered

No response

Importance

would make my life easier

Additional Information

No response

[jeffchuber]

@washedPat thanks for submitting the issue. I would definitely try handling memory. Do you know how Fly handles scaling containers? Chroma is currently designed to be single node (though this will change soon) and multiple nodes won't be able to share memory. Just something to keep in mind!

[jeffchuber]

@washedPat closing this for now. happy to re-open. we are working a refactor and will think about broader deployments after that lands.

[testA113]

Hello @washedPat! I just got a hacky version of chroma server working on fly.io :) This hacky version adds the chroma database at build time, so please don't push this image publicly if you don't want people to have access to your DB.

Hacky version

Dockerfile:

FROM chromadb/chroma:0.5.0 as chroma

# copy the chroma_db directory from local directory to the chroma image
# TODO: this is a temporary solution, we should use a volume instead using a startup script
COPY ./chroma_db /chroma/chroma

fly.toml

app = '<your app name>'
primary_region = 'syd'

[build]

[env]
  IS_PERSISTENT = 'TRUE'
  PERSIST_DIRECTORY = '/chroma/chroma' 
  ANONYMIZED_TELEMETRY = 'FALSE'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 1
  processes = ['app']

[[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 1

In the same directory, run fly deploy (you already have the toml file, so you don't need to run fly launch)

This should get chroma running at an address like https://your-app-name.fly.dev . You can test this out by going to https://your-app-name.fly.dev/docs .

Note that fly automatically makes this https, and that you don't need to add the 8000 port (your connecting on port 443 instead).

If you want your client app to talk to it. Use a configuration similar to this (llamaindex):

chroma_client = chromadb.HttpClient(host="<your-app-name>.fly.dev", port=443, ssl=True)

Future version

In future, I'm keen to mount the db files as a volume in fly.io, so that I can simply use the chroma-db image. Health checks would also be a nice addition.