ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Fix timeout and output parsing issues for local models

Open jjmachan opened this issue 7 months ago • 0 comments

This PR addresses issue #2044 by improving the handling of local models in Ragas.

Changes

  1. Added a new is_local_model flag to RunConfig with increased timeout (10 minutes) for local models
  2. Improved the JSON extraction logic in extract_json to handle common issues with local model outputs:
    • Better handling of markdown code blocks
    • Converting single quotes to double quotes
    • Removing trailing commas in arrays and objects
  3. Enhanced the RagasOutputParser to be more robust with malformed JSON
  4. Added tests to verify the fixes

These changes should help users who are using local models like Ollama to get better results with metrics like context_recall, faithfulness, and context_precision by reducing timeouts and improving output parsing.

Fixes #2044

@jjmachan can click here to continue refining the PR

jjmachan avatar May 17 '25 17:05 jjmachan