llama-stack icon indicating copy to clipboard operation
llama-stack copied to clipboard

[rag evals][1/n] refactor base scoring fn & data schema check

Open yanxi0830 opened this issue 10 months ago • 0 comments

What does this PR do?

  • Refactor BaseScoringFn to be just a minimal interface, add new RegistrableBaseScoring
  • Refactor data schema check
    • To separately evaluate retrieval component in RAG, we will have scoring functions needing "context" column additionally.
  • Refactor braintrust eval (more scoring fn added & tested in following PR)

Test Plan

pytest -v -s -m llm_as_judge_scoring_together_inference scoring/test_scoring.py --judge-model meta-llama/Llama-3.2-3B-Instruct
pytest -v -s -m basic_scoring_together_inference scoring/test_scoring.py
pytest -v -s -m braintrust_scoring_together_inference scoring/test_scoring.py
image
pytest -v -s -m meta_reference_eval_together_inference eval/test_eval.py
pytest -v -s -m meta_reference_eval_together_inference_huggingface_datasetio eval/test_eval.py
image

Before submitting

  • [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • [ ] Ran pre-commit to handle lint / formatting issues.
  • [ ] Read the contributor guideline, Pull Request section?
  • [ ] Updated relevant documentation.
  • [ ] Wrote necessary unit or integration tests.

yanxi0830 avatar Dec 19 '24 22:12 yanxi0830