ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Fix issue #2044: Improve output parsing and timeout for local LLMs

Open jjmachan opened this issue 7 months ago • 0 comments

This PR addresses the issues reported in #2044 where local LLMs like Ollama with qwen2.5:14b were encountering RagasOutputParserException and TimeoutError during evaluation.

Changes

  1. Increased default timeout in RunConfig from 180 to 300 seconds to accommodate slower local LLMs
  2. Enhanced extract_json function to better handle various LLM output formats:
    • Added support for markdown code blocks
    • Added handling for single quotes in JSON
    • Added handling for trailing commas in JSON
    • Improved JSON extraction from text with surrounding content
  3. Improved RagasOutputParser with:
    • More robust error handling
    • Increased default retries from 1 to 3
    • Better error messages with details about the failure
    • Added suggestions for working with local LLMs in error messages

Added tests

  • Tests for extract_json with various input formats
  • Tests for RagasOutputParser fallback mechanism
  • Tests for timeout configuration

Fixes #2044

@jjmachan can click here to continue refining the PR

jjmachan avatar May 17 '25 17:05 jjmachan