langgenius
Implementation of Automatic fallback Mechanism for Models During Service Interruptions
Self Checks
- [X] I have searched for existing issues search for existing issues, including closed ones.
- [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
- [X] Please do not modify this template :) and fill in all the required fields.
1. Is this request related to a challenge you're experiencing? Tell me about your story.
We have been using the 'sonnet-3.5' model as our primary model for generating content. However, we've been experiencing frequent instabilities and failures, primarily due to rate limits. These issues disrupt our workflow and affect the reliability of the service. To address this, we propose implementing an automatic fallback option to a secondary model, such as 'gpt-4o'. This would allow us to maintain continuity and minimize downtime during interruptions with the primary model. Considering the block-based visual interface of Dify, a mechanism that triggers an action like "if fails, go to..." or generates a variable upon a block failure could be very useful. The main issue currently is that if a block fails, all processing is lost. This is particularly problematic with newer LLMs and smaller servers, which often experience some inconsistencies. Therefore, implementing this feature would be highly welcomed.
2. Additional context or comments
Incorporating an automatic fallback mechanism would significantly enhance the system's robustness and reliability. Given the critical nature of our applications, having a secondary model ready to handle requests when the primary model encounters issues would ensure a seamless user experience. Furthermore, the ability to switch between different model providers (e.g., from OpenAI to Anthropic) as needed would be highly beneficial. Implementing a feature that triggers a fallback action in case of a block failure would also be valuable, as current block failures result in the loss of all processing.
3. Can you help us with this feature?
- [X] I am interested in contributing to this feature.
I think not only for the LLM node but also other types of node as well. eg, if HTTP request fails, it should retry another service or search things in KB. I have already forward this feature to our PM last week.