marvin icon indicating copy to clipboard operation
marvin copied to clipboard

large memory footprint introduced by `chromadb` default embeddings

Open zzstoatzz opened this issue 1 year ago • 1 comments

ChromaDB uses sentence-transformers by default for embeddings, which requires torch, which all together makes the footprint of chroma something like ~5GB.

We use OpenAI's text-embedding-ada-002 model offered via chromadb.utils.embedding_functions, but chromadb enforces the sentence-transformers dependency at this time.

We should find a way around this if chroma doesn't make sentence-transformers an optional dep

zzstoatzz avatar Mar 28 '23 17:03 zzstoatzz

#87 makes chromadb an optional extra, but will keep this open as it seems chroma is likely to make sentence-transformer optional

zzstoatzz avatar Mar 28 '23 20:03 zzstoatzz

@zzstoatzz chroma 0.4.0 is much smaller now.

jeffchuber avatar Jul 20 '23 21:07 jeffchuber