promptfoo feat: ability to set evaluateOptions.outputVars [WIP]

Summary by Sourcery

Add ability to configure and filter output variables in evaluation results

New Features:

Introduce outputVars option to specify which variables to include in evaluation results
Allow configuring output variables through config file, CLI, or environment variable

Enhancements:

Modify result processing to support selective variable output
Add flexibility in controlling which test variables are displayed in results

May 02 '25 17:05 typpo

Reviewer's Guide

This pull request introduces the evaluateOptions.outputVars option, allowing users to specify which test case variables should be included in the evaluation results and in what order. The option is defined in the configuration file (promptfooconfig.yaml) or via the PROMPTFOO_OUTPUT_VARS environment variable. It's merged with command-line options in doEval, passed through the evaluation pipeline (evaluate, Evaluator, runEval), stored in the result metadata, and used during results processing (EvalResult.toJSON, convertResultsToTable) and UI rendering (ResultsView, store) to filter and order the output variables.

File-Level Changes

Change	Details	Files
Added `outputVars` option to `EvaluateOptions` and configuration handling.	Defined `outputVars` as an optional string array in `EvaluateOptionsSchema`. Merged `evaluateOptions` from the loaded configuration file with existing options in `doEval`. Added `outputVars` example to the sample `promptfooconfig.yaml`.	`src/types/index.ts` `src/commands/eval.ts` `examples/simple-cli/promptfooconfig.yaml` `src/util/config/load.ts`
Propagated `evaluateOptions` through the evaluation pipeline and stored it in result metadata.	Passed the merged `evaluateOptions` to the `evaluate` function. Stored `evaluateOptions` in the `Evaluator` class instance. Added `evaluateOptions` (specifically `outputVars`) to the `metadata.config` field within the `EvaluateResult` created in `runEval`.	`src/commands/eval.ts` `src/evaluator.ts` `src/models/evalResult.ts`
Implemented logic to filter and order variables based on `outputVars` during results processing.	Added `PROMPTFOO_OUTPUT_VARS` environment variable definition. In `EvalResult.toJSON()`, read `outputVars` from metadata (`config.evaluateOptions.outputVars`) or the `PROMPTFOO_OUTPUT_VARS` environment variable to filter/order the `vars` property. Modified `convertResultsToTable` to accept an optional `outputVars` parameter to filter/order the table header and row variables. Updated `Eval.resultsTable()` to pass `outputVars` from its config to `convertResultsToTable`.	`src/models/evalResult.ts` `src/util/convertEvalResultsToTable.ts` `src/models/eval.ts` `src/envars.ts`
Updated UI components to correctly handle results potentially filtered by `outputVars`.	Updated `setTableFromResultsFile` in the store to potentially use `convertResultsToTable` with `outputVars` from the results file config. Adjusted comparison logic in `ResultsView` to potentially handle tables with different variable sets due to `outputVars`.	`src/app/src/pages/eval/components/ResultsView.tsx` `src/app/src/pages/eval/components/store.ts`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an issue from a review comment by replying to it. You can also reply to a review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull request title to generate a title at any time. You can also comment @sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in the pull request body to generate a PR summary at any time exactly where you want it. You can also comment @sourcery-ai summary on the pull request to (re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the pull request to resolve all Sourcery comments. Useful if you've already addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull request to dismiss all existing Sourcery reviews. Especially useful if you want to start fresh with a new review - don't forget to comment @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

May 02 '25 17:05 sourcery-ai[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	14b19a9ea80780374c6a8ac1bf13523dc4e89991	✅ Finished

Files

File	Pull Request
src/app/src/pages/eval/components/store.ts	❌ Failure (I failed to write the unit tests for the file.)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	14b19a9ea80780374c6a8ac1bf13523dc4e89991	✅ Finished

Files

File	Pull Request
src/util/config/load.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/util/convertEvalResultsToTable.ts	🟣 Merged https://github.com/promptfoo/promptfoo/pull/3876
src/types/index.ts	🟣 Merged https://github.com/promptfoo/promptfoo/pull/3874
src/models/eval.ts	🚫 Skipped (There's no need to update the test code)
src/commands/eval.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/models/evalResult.ts	🟣 Merged https://github.com/promptfoo/promptfoo/pull/3875
src/envars.ts	🚫 Skipped (There's no need to update the test code)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	6d49026d6e7f14d9c568b5284f68b2ae7519dcf6	✅ Finished

Files

File	Pull Request
src/util/config/load.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/util/convertEvalResultsToTable.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/models/evalResult.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/models/eval.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/envars.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)
src/commands/eval.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	6d49026d6e7f14d9c568b5284f68b2ae7519dcf6	✅ Finished

Files

File	Pull Request
src/app/src/pages/eval/components/store.ts	🛑 Cancelled (Canceled by Auto Rebase Detail)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	91c032286304921f73e25c57bdd0430b215bb259	✅ Finished

Files

File	Pull Request
src/util/config/load.ts	🔴 Closed https://github.com/promptfoo/promptfoo/pull/3878
src/util/convertEvalResultsToTable.ts	🚫 Skipped (There's no need to update the test code)
src/models/evalResult.ts	🚫 Skipped (There's no need to update the test code)
src/models/eval.ts	🚫 Skipped (There's no need to update the test code)
src/types/index.ts	🚫 Skipped (There's no need to update the test code)
src/envars.ts	🚫 Skipped (There's no need to update the test code)
src/commands/eval.ts	❌ Failure (Failed to solve the problem.)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	91c032286304921f73e25c57bdd0430b215bb259	✅ Finished

Files

File	Pull Request
src/app/src/pages/eval/components/store.ts	❌ Failure (I failed to write the unit tests for the file.)

[!TIP] You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

May 02 '25 17:05 gru-agent[bot]