[Feature]: Generic Fallback API

[CircArgs]

The Feature

Users should be able to provide a callback in the case of an exception to handle fallingback or however else they want to handle the issue.

Motivation, pitch

Currently, there are decent defaults to handle fallbacks, yet there have been many issues reported about the fallbacks and a callback interface would allow users more freedom and control. Further, the fallback design could be pythonic where methods of fallback could be reimplemented not as hard logic but as swappable elements provided by default such as a class TokenLimitFallback which when passed looks for exceptions related to too many tokens. Similarly, the defaults could be implemented for rate limits, etc and then the user would follow a similar pattern for custom handlers.

[krrishdholakia]

i'm confused - you can already handle fallbacks on the proxy https://docs.litellm.ai/docs/proxy/reliability

What are you referring to?

[ishaan-jaff]

@CircArgs are you trying to do this on the litellm proxy or SDK ?

[krrishdholakia]

Hey @CircArgs

closing as you can now write your own routing strategy - https://docs.litellm.ai/docs/routing#advanced---routing-strategies-%EF%B8%8F

If this doesn't solve your problem, can we do a 10min call to discuss further?

https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat