ragas
ragas copied to clipboard
Fix issue #2044: Improve output parsing and timeout for local LLMs
This PR addresses the issues reported in #2044 where local LLMs like Ollama with qwen2.5:14b were encountering RagasOutputParserException and TimeoutError during evaluation.
Changes
- Increased default timeout in RunConfig from 180 to 300 seconds to accommodate slower local LLMs
- Enhanced extract_json function to better handle various LLM output formats:
- Added support for markdown code blocks
- Added handling for single quotes in JSON
- Added handling for trailing commas in JSON
- Improved JSON extraction from text with surrounding content
- Improved RagasOutputParser with:
- More robust error handling
- Increased default retries from 1 to 3
- Better error messages with details about the failure
- Added suggestions for working with local LLMs in error messages
Added tests
- Tests for extract_json with various input formats
- Tests for RagasOutputParser fallback mechanism
- Tests for timeout configuration
Fixes #2044
@jjmachan can click here to continue refining the PR