promptulate icon indicating copy to clipboard operation
promptulate copied to clipboard

Add retry mechanism to LLMFactory and BaseLLM

Open Undertone0809 opened this issue 6 months ago • 1 comments

Related to #651

Add retry mechanism to LLMFactory and LiteLLM classes to increase robustness.

  • LLMFactory:

    • Add max_retry parameter to LLMFactory.build method in promptulate/llms/factory.py.
    • Pass max_retry parameter to LiteLLM constructor.
  • LiteLLM:

    • Add max_retry attribute to LiteLLM class in promptulate/llms/_litellm.py.
    • Modify _predict method to include retry logic.
  • AIChat:

    • Add max_retry parameter to AIChat constructor in promptulate/chat.py.
    • Pass max_retry parameter to LLMFactory.build method.
  • Tests:

    • Add tests in tests/llms/test_factory.py to verify retry mechanism.
    • Mock LiteLLM to simulate failure and retry behavior.

For more details, open the Copilot Workspace session.

Summary by CodeRabbit

  • New Features

    • Introduced a max_retry parameter across various functionalities, enhancing user control over retry attempts during LLM initialization and chat operations.
    • Improved error handling and robustness in the LiteLLM prediction process with a retry mechanism.
  • Bug Fixes

    • Enhanced resilience in LLM interactions by implementing a retry logic for failed attempts.
  • Tests

    • Added new tests to validate the retry mechanisms for the LiteLLM model, ensuring robust error handling.

Undertone0809 avatar Jul 29 '24 18:07 Undertone0809