promptfoo
promptfoo copied to clipboard
promptfoo
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command...
Big Changes: - New Table: https://github.com/promptfoo/promptfoo/pull/1776/files#diff-660e00170b1266d7afb7e2676c50217c2ac27bb283dff59b0e82660daf775307R1-R25 - New output format: [examples/simple-cli/output.json](https://github.com/promptfoo/promptfoo/pull/1776/files#diff-e0bb0dfb8b7d049e5eee6104980e31cad839073570ed090b1743b82a02cb8751R2) Normalize the eval results by storing them in their own table. - Added a new `Eval`, `EvalResults` and `Provider`...
# Questions: 1. When I am creating an output file via `pf eval -c --output `, the result always has a version of **2**. Is this a bug or desired...
**Describe the bug** From this [documentation page](https://www.promptfoo.dev/docs/configuration/expected-outputs/python/#external-py), it seems quite straightforward to read config parameters passed to an external python script for evaluations. Unfortunately the context object does not seem...
**Describe the bug** When vars contain a complex map of nested values, reading content of files with file:// prefix doesn't seem to work. **To Reproduce** Here, the prompt template is...
**Describe the bug** Unable to generate share URL . Share button keeps showing infinite processing **To Reproduce** Steps to reproduce the behavior, including example Promptfoo configurations if applicable: 1. Run...
This plugin introduces a Math-Prompt attack, where harmful instructions are encoded as mathematical problems. The goal is to trick large language models (LLMs) into proving or solving the encoded instructions,...
Red team tests are now stored in a separate file and referenced in the main configuration, rather than being embedded directly in the existing promptfooconfig or promptfooconfig being copied into...
# Implement `promptfoo push` and Remote Prompts This PR introduces the ability to save and reference prompts in a central location, addressing the needs of users on larger teams. ##...
### Notes: * Update `CallApiContextParams` to include optional gateway object * Uses a single gateway object created for each eval run * Can handle multiple requests concurrently when running with...
