BerriAI
completion call with `fallbacks` using Python SDK seems to be stuck in an infinite loop
The below code snippet seems to result in an infinite loop of trying claude-3-5-sonnet-20241022, gpt-4o-2024-11-20, claude-3-5-sonnet-20241022, ...
Setting num_retries or max_fallbacks does not solve the issue.
from litellm import completion
resp = completion( model="claude-3-5-sonnet-20241022", messages=messages, api_key="wrong_api_key", fallbacks=[ { "model": "gpt-4o-2024-11-20", "api_key": "wrong_key_2", "organization": "some_org", } ] )
Additionally, reading the code of completion_with_fallbacks:
if isinstance( model, dict ): # completion(model="gpt-4", fallbacks=[{"api_key": "", "api_base": ""}, {"api_key": "", "api_base": ""}]) kwargs["api_key"] = model.get("api_key", None) kwargs["api_base"] = model.get("api_base", None) model = model.get("model", original_model)
It looks like we are missing extracting organization and api_version. Does this mean falling back from claude -> openai (which requires organization) or from openai -> azure is not possible?
