ERROR from google generative AI = 504 Deadline Exceeded

[SumeetSandhu]

Hi, I'm running Jupyter NB locally (conda), running chroma on it and storing collections locally (chroma_client = chromadb.PersistentClient(path="path/chroma")).

I am able successfully save the collection and get it after restarting NB for a toy example of 10 short documents (url, 1 sentence, etc).

However when I try to do this for 3000 such documents, I get an error from google generative AI: 504 DEADLINE_EXCEEDED

Any quick tips on how to adjust "client-provided deadline"? (see https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/api-errors)

[tazarov]

hey @SumeetSandhu, are you using Google Vertex Embedding Function from Chroma? If that is the case, how do you chunk or batch your documents? Is it possible that you try to send too big of a payload to Google Vertex AI API endpoint?

[SumeetSandhu]

thanks for responding @tazarov I'm using Google Gemini as embedding function https://docs.trychroma.com/embeddings/google-gemini. You are right - this has to do with batching documents.

I tried adding all the documents to the collection at once - and got this error. What did work is adding documents one at a time - but occasionally that still gives an error so I added a try/except loop that sleeps for 30-60 seconds if it fails, before re-trying.

thanks - I think this is the best we can do at this time :)

[tazarov]

@SumeetSandhu, I can understand the challenge with this. APIs are fickle and sometimes fail due to load, global rate limit, user rate limit, and a number of other reasons.

If you haven't already, have a look at https://github.com/jd/tenacity. You can wrap it around your batch embedding to retry (it supports exponential backoff, amongst other strategies)