Ian Webster

We actually support this through the `--filter-providers` flag on `eval`. It takes a regex which matches on `id` and `label`. Here's the implementation: https://github.com/promptfoo/promptfoo/blob/main/src/commands/eval/filterProviders.ts#L3 `targets` is just an alias for...

hmm, it _should_ match the label as well: https://github.com/promptfoo/promptfoo/blob/main/src/commands/eval/filterProviders.ts#L17

Out of curiosity, how large is the eval you are trying to share? Are you able to share an example (e.g. produced by `promptfoo eval -o output.json`)

Unfortunately we can't support payloads of that size via public sharing right now. You could try self-hosting https://www.promptfoo.dev/docs/usage/self-hosting/ or get in touch for our paid cloud version: https://www.promptfoo.dev/contact/

One more idea: you can set `PROMPTFOO_LIGHTWEIGHT_RESULTS=1` to strip some information from the eval result and make it much smaller. This may make it easier to share :)

In progress at #2709

cool strat

Thanks for flagging. This is a RAGAS metric that copied their implementation. I think they've since refined it, and it probably makes sense to re-implement the metrics according to what...

Hi @brendanator, we actually show average cost per test case already! It's tucked away in that gray text at the bottom of the header

You're correct, batch inference is not supported right now. We don't have specific plans to implement this, although we would gladly accept a PR. On Mon, Dec 16, 2024 at...