Rate Limit feature for Azure/OpenAI results generation

[Jeff-ZYX]

Is your feature request related to a problem? Please describe. Current architecture doesn't seem to support API instances with inbuilt rate limits, causing errors in generation

Describe the solution you'd like A sleep function that can be configured to change rates of generation of GPT completions.

[chakravarthik27]

Hi @Jeff-ZYX

We acknowledge and appreciate the information you brought to our attention. Currently, we do not have a built-in rate restriction capability in our architecture for API instances. We understand that controlling the rate of generation is crucial to prevent errors and ensure error-free operation for GPT completions. We thank you for recommending the addition of a programmable sleep function to efficiently regulate the generating rate. We value your input and will prioritize this feature request for future updates, even though we are unable to offer a quick fix.