promptfoo icon indicating copy to clipboard operation
promptfoo copied to clipboard
verified publisher icon promptfoo

feat(eval): add global pass rate threshold

Open mldangelo opened this issue 1 year ago • 1 comments

This PR addresses #1441 It introduces a configurable pass rate threshold for evals. The command now exits with PROMPTFOO_FAILED_TEST_EXIT_CODE if the pass rate falls below the specified threshold. When the threshold is not set, the existing behavior (0 failures allowed) is maintained.

  • Add PROMPTFOO_PASS_RATE_THRESHOLD environment variable (default: 100%)
  • Modify exit behavior to consider the pass rate threshold
  • Update documentation to reflect new functionality

CC @MatanYadaev

mldangelo avatar Aug 14 '24 22:08 mldangelo

Amazing @mldangelo ! WDYT about allowing the user to configure the pass rate threshold inside the configuration, instead of in a environment variable? In my use case, I have many different promptfoo configurations, each one has different pass rate threshold.

MatanYadaev avatar Aug 14 '24 22:08 MatanYadaev

Amazing @mldangelo! WDYT about allowing the user to configure the pass rate threshold inside the configuration, instead of in an environment variable? In my use case, I have many different promptfoo configurations, each one has a different pass rate threshold.

@MatanYadaev This is a great idea! Luckily, it is already supported! In your promptfoo config files, you can set:

env:
  PROMPTFOO_PASS_RATE_THRESHOLD: 76.54

Thanks again for the product feature suggestion.

mldangelo avatar Aug 19 '24 04:08 mldangelo