[ENH]: OpenAI Rate Limiting

[johnisanerd]

What happened?

In Python, I ran a collection.add on a set of documents about ~10,000. Our account is rate limited to 10,000 RPM. We kept getting rate limited when we added a large set of documents.

We solved this by slicing up the list, and adding in a crude sleep. A more elegant solution would be greatly appreciated!

Versions

chromadb==0.4.24 Python 3.11.6

Relevant log output

Don't have any log output.

[nicolasgere]

Hi @johnisanerd, that's a great feature idea, and it could indeed be a part of Chroma. We don't have immediate plans to work on it in the short term, but it's on our radar for future consideration. Thank you

[johnisanerd]

Is there a cookbook or somewhere I could contribute a snippet of code? I made a workaround example that requires a little bit of finesse, I could share.

[tazarov]

@johnisanerd, you can either submit a PR with your proposal or add it here if you want us to discuss it. We should solve this problem for all EFs, not just OpenAI. Many embedding functions rely on external API calls and are susceptible to network issues, rate limiting, etc.