promptfoo

promptfoo copied to clipboard

Azure model o1-mini returns error max_tokens is not supported

2

When I use the azure:chat:o1-mini model with Promptfoo v0.111.0, I get the following error: > API response error: unsupported_parameter > Unsupported parameter: 'max_tokens' is not supported with this model. Use...

mariana-pineda

chore: Send auth info to cloud

4

sklein12

## Trigger Info | Trigger Type | Triggered By | Source Pull Request | Assignment | | ------------ | ------------ | ------------------- | ---------- | | Ready for Review |...

gru-agent[bot]

## Trigger Info | Trigger Type | Triggered By | Source Pull Request | Assignment | | ------------ | ------------ | ------------------- | ---------- | | Ready for Review |...

gru-agent[bot]

Custom HTTP grader gets empty string in request body for llm-rubric eval

2

**Describe the bug** I want to use a custom HTTP provider as a grader in an LLM-rubric evaluation, the endpoint is successfully called, however the `{{rubric}}` variable is resolved as...

vira-s

Hi. I've installed promptfoo form tthis package https://pypi.org/project/promptfoo/. Running promptfoo ends with the following exception: ```fish (promptfoo-self-learning-py3.13) damian@dgiebas-ubu-i4b-pl ~/A/promptfoo-self-learning (master)> promptfoo --version Traceback (most recent call last): File "/home/damian/.cache/pypoetry/virtualenvs/promptfoo-self-learning-nABErWj3-py3.13/bin/promptfoo", line...

Draqun

refactor(providers): load default providers from a registry

2

This refactor centralizes the logic for loading default providers, aligning it with their respective implementations. - Moved default provider configuration logic to individual provider files. - Updated provider selection logic...

mldangelo

Hello, Is it possible to read paths from an environment variable in the promptfooconfig.yaml? As you can see in the following config.yaml, I read prompt, provider, assertions from path "../../test"....

merveozbay

Guardrails server

1

Hey! Looking at this file: https://github.com/promptfoo/promptfoo/blob/main/src/guardrails.ts I noticed the guardrails service is sending requests to a hosted API. Could you clarify: - Where is the actual guardrails server hosted? -...

rafaelsandroni

Prompt injection guardrail false positives

1

**Describe the bug** Almost every simple message gets flagged as a prompt injection attack. **Screenshots** **System information:** ``` { "version": "0.104.0", "platform": { "os": "darwin", "release": "23.6.0", "arch": "arm64", "nodeVersion":...

manankalra