promptfoo
promptfoo copied to clipboard
promptfoo
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command...
Currently I have code like this, hardcoding the model cost into my custom provider, and returning the total cost that way. ```javascript // ... const inputCost = 0.00025 / 1000;...
Thank you for adding #765 (assert-set) and #903 (__threshold for csv). 👍 But I found an issue while using it. I thought below should be shown same behavior. a.yaml ```...
We provide a rendered view of the output (as markdown) in the output payload. Would it be possible to have an expression to select that subfield and render it instead...
So that users can put there system prompt here. Ref. to the [document](https://ai.google.dev/api/rest/v1beta/models/streamGenerateContent#request-body). Note that the `systemInstruction` types `Content`, in form of `{ parts: [ { text: ... } ]...
Currently, all provider config must be done at the top-level. It's not possible to run the same provider with different configuration. It should be possible to override the provider config...
Changing the layout of the eval result table in the web UI causes the result cells to re-render markdown: https://github.com/promptfoo/promptfoo/assets/17098959/c22fc062-fc05-4ff3-9c31-6106a146618a Interestingly it doesn't look like that happens for the variable...
I wanted to share a link to my team with the evaluation done and my previous evaluations of the prompts as well. However when I click on Evals/prompts or any...
The image previewer is awesome! However, I'm unable to preview the image if it's a variable rather than a result. I'm also unable to preview the image if there's an...
Hi, I tried to use this tool using the provider `vertex:text-bison@002` as the config file below: ``` # Learn more: https://promptfoo.dev/docs/configuration/guide description: 'Testing' prompts: - 'Convert this English to {{language}}:...
When rendering with markdown in the web viewer, if I manually fail the test case (i.e. thumbs down), the markdown appears _and_ is rendered: https://github.com/promptfoo/promptfoo/assets/17098959/a80c214c-511b-449e-aea8-eb45bab074cc Is this intended? Would be...