text-embeddings-inference

text-embeddings-inference copied to clipboard

Feature request

Does it make sense for TEI to add a cache layer for embeddings?

Not sure if TEI supports this already.

If not, I'd be curious if it makes sense to add one. This raises some other questions: where should that live, how big should it be, etc

Motivation

Even though some models have large vocabs, I think a cache makes sense where some strings repeat often, like in code models.

Your contribution

No rust experience, but I can help with anything else