Add a non-blocking CI report for sample outputs (PromptProof)

[geminimir]

Describe the feature or improvement you're requesting

I’d like to add a minimal report-only GitHub Action that replays one recorded run and publishes a one-page HTML report per PR. It will:

• Trigger only when the three new files change (no impact on the example code).
• Validate { text: string } and forbid obvious PII via regex.
• Show cost/latency for the recorded run.
• Use seed + runs=3 for stability; fixtures only, no live calls.

Here are the three files:

Files to add

.github/workflows/promptproof.yml

name: PromptProof

on:
  pull_request:
    paths:
      - ".github/workflows/promptproof.yml"
      - "promptproof.yaml"
      - "fixtures/promptproof/**"

jobs:
  proof:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: geminimir/promptproof-action@v0
        with:
          config: promptproof.yaml
          baseline-ref: ${{ github.event.pull_request.base.sha }}
          runs: 3
          seed: 1337
          max-run-cost: 0.20
          report-artifact: promptproof-report
          mode: report-only

promptproof.yaml

mode: fail
format: html

fixtures:
  - path: fixtures/promptproof/quickstart.json

checks:
  - id: sample_schema
    type: schema
    json_schema:
      type: object
      properties:
        output:
          type: object
          properties:
            text: { type: string, minLength: 1 }
          required: [text]
      required: [output]

  - id: forbid_emails
    type: regex_forbid
    pattern: "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"

budgets:
  max_run_cost: 0.20

stability:
  runs: 3
  seed: 1337

fixtures/promptproof/quickstart.json

{
  "record_id": "openai-qs-001",
  "input": { "prompt": "Say hi" },
  "output": { "text": "Hi! Deterministic sample output." }
}

What maintainers get

A single HTML report artifact per PR (schema/regex/cost summary). Zero live calls; easy to delete if unwanted. References Sample report: https://geminimir.github.io/promptproof-action/reports/before.html

If this sounds okay, I’ll open a 3-file PR and can tweak the checks/paths to your preference.

Additional context

Marketplace: https://github.com/marketplace/actions/promptproof-eval Demo project: https://github.com/geminimir/promptproof-demo-project