ragas
ragas copied to clipboard
Fix timeout and output parsing issues for local models
This PR addresses issue #2044 by improving the handling of local models in Ragas.
Changes
- Added a new
is_local_modelflag toRunConfigwith increased timeout (10 minutes) for local models - Improved the JSON extraction logic in
extract_jsonto handle common issues with local model outputs:- Better handling of markdown code blocks
- Converting single quotes to double quotes
- Removing trailing commas in arrays and objects
- Enhanced the
RagasOutputParserto be more robust with malformed JSON - Added tests to verify the fixes
These changes should help users who are using local models like Ollama to get better results with metrics like context_recall, faithfulness, and context_precision by reducing timeouts and improving output parsing.
Fixes #2044
@jjmachan can click here to continue refining the PR