genkit icon indicating copy to clipboard operation
genkit copied to clipboard

Checks testapp for evaluators

Open HunterHeston opened this issue 1 year ago • 1 comments

Just adding a test application which configures the Checks evaluators.

run genkit eval:run input-file.json --evaluators=checks/dangerous_content,checks/pii_soliciting_reciting,checks/harassment,checks/sexually_explicit,checks/hate_speech,checks/medical_info,checks/violence_and_gore,checks/obscenity_and_profanity

Where input-file.json is a file of this format:

[
  {
    "testCaseId": "test_case_id_1",
    "input": "The input to your model.",
    "output": "Example model output which. This is what will be evaluated."
  }
]

Checklist (if applicable):

  • [ ] Tested (manually, unit tested, etc.)
  • [ ] Docs updated

HunterHeston avatar Nov 09 '24 00:11 HunterHeston

I am consolidating all evaluator testapps into one... https://github.com/firebase/genkit/pull/1294

Since Checks is not very different from existing evals, can we just reuse this testapp instead? You can add a separate Checks evaluator section to the README file.

ssbushi avatar Nov 14 '24 15:11 ssbushi