Rate Limited error

[lancerguy]

openai.error.RateLimitError: Rate limit reached for 10KTPM-200RPM in organization org-fK5bb25UFhVbebfBtfCejGc4 on tokens per min. Limit: 10000 / min. Please try again in 6ms. Contact us through our help center at help.openai.com if you continue to have issues.

Maybe a way to resume so all the runtime isn't just lost?

[wangxiaochun520]

sample solve way: async def _achat_completion_stream(self, messages: list[dict]) -> str: # response = await openai.ChatCompletion.acreate(**self._cons_kwargs(messages), stream=True) while True: try: response = await openai.ChatCompletion.acreate( **self._cons_kwargs(messages), stream=True ) break except Exception as e: print(f"Error {e}, sleep 60s") time.sleep(60) continue

[ameramayreh]

sample solve way: async def _achat_completion_stream(self, messages: list[dict]) -> str:

response = await openai.ChatCompletion.acreate(**self._cons_kwargs(messages), stream=True)

while True: try: response = await openai.ChatCompletion.acreate( **self._cons_kwargs(messages), stream=True ) break except Exception as e: print(f"Error {e}, sleep 60s") time.sleep(60) continue

Thanks, it looks it is working.

[ashishawasthi]

Please review pull request https://github.com/geekan/MetaGPT/pull/213 to retry (with exponential backoff) if OpenAI rate limit is hit. Please leave your comments.

[voidking]

same as #206

[krrishdholakia]

hey @ashishawasthi, I'm the maintainer of LiteLLM we allow you to create a Router to maximize throughput by load balancing multiple azure/openai instances

I'd love to get your feedback if this solves your issue

Here's the quick start

from litellm import Router

model_list = [{ # list of model deployments 
    "model_name": "gpt-3.5-turbo", # model alias 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "azure/chatgpt-v-2", # actual model name
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE")
    }
}, {
    "model_name": "gpt-3.5-turbo", 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "azure/chatgpt-functioncalling", 
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE")
    }
}, {
    "model_name": "gpt-3.5-turbo", 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "gpt-3.5-turbo", 
        "api_key": os.getenv("OPENAI_API_KEY"),
    }
}]

router = Router(model_list=model_list)

# openai.ChatCompletion.create replacement
response = await router.acompletion(model="gpt-3.5-turbo", 
                messages=[{"role": "user", "content": "Hey, how's it going?"}])

print(response)