Skip to content
TruthVouch Docs

Promptfoo Integration

Integrate Promptfoo (the leading open-source LLM evaluation and red-teaming tool) with TruthVouch to automatically convert eval results into compliance evidence records. Turn your LLM evaluations into governance artifacts that satisfy regulatory frameworks like the EU AI Act, NIST AI RMF, and ISO 42001.

Value proposition: Already using Promptfoo? Pipe your eval results into TruthVouch for centralized compliance tracking and obligation mapping.

Prerequisites

  • TruthVouch account — Sign up at app.truthvouch.ai
  • Professional tier or higher — Starter tier does not include compliance evidence import
  • API key — Generate from Settings > API Keys in the TruthVouch dashboard
  • TruthVouch CLI — Version 0.2.0 or later:
    Terminal window
    npm install -g @truthvouch/cli
  • Promptfoo — Version 0.40.0 or later:
    Terminal window
    npm install -g promptfoo

Quick Start

Step 1: Run your Promptfoo evaluation

Terminal window
# Create or use an existing promptfoo config
promptfoo eval -c promptfooconfig.yaml -o results.json

Step 2: Import results to TruthVouch

Terminal window
# First, authenticate with TruthVouch
truthvouch login --api-key vt_live_your_api_key


# Then import the results
truthvouch import promptfoo results.json

That’s it! Your eval results are now compliance evidence records in TruthVouch, automatically linked to relevant obligations.

Step 3: View in dashboard

Log in to app.truthvouch.ai and navigate to Compliance > Evidence to see your Promptfoo results alongside other evidence sources.

How It Works

Evidence Mapping

Promptfoo results are classified into compliance evidence types based on their findings:

Promptfoo SignalEvidence TypeFramework HintsObligation Keywords
Red-team failures (harmful:*, pii:*, hijacking, overreliance)Security evidenceEU AI Act, NIST AI RMF, ISO 42001security testing, adversarial robustness, risk management
Low faithfulness (< 0.8)Accuracy evidenceEU AI Act, ISO 42001accuracy, output quality, performance monitoring
High toxicity (> 0.2)Fairness evidenceEU AI Act, NIST AI RMFbias, fairness, non-discrimination
Passing testsGeneral testing evidenceISO 42001, SOC 2testing, quality assurance, continuous monitoring

Idempotency

Re-importing the same Promptfoo JSON file is safe — TruthVouch uses SHA-256 idempotency keys to prevent duplicates. If you import the same file twice:

  • First import: Creates summary + per-failure evidence records
  • Second import: All records are skipped (zero new evidence created)
  • Updated results: New failures are imported; passing tests remain as-is

This ensures your compliance audit trail is accurate without manual deduplication.

CLI Reference

Basic import

Terminal window
truthvouch import promptfoo results.json

With AI system linking

Link the evidence to a specific AI system in your inventory:

Terminal window
truthvouch import promptfoo results.json --ai-system-id abc-123-uuid

With labels

Attach metadata labels for filtering:

Terminal window
truthvouch import promptfoo results.json \
  --label env=production \
  --label branch=main \
  --label model=gpt-4o

Machine-readable output

For CI/CD piping, use --json to get structured output:

Terminal window
truthvouch import promptfoo results.json --json | jq '.evidenceIds'

Full command reference

Usage: truthvouch import promptfoo [options] <file>


Import Promptfoo eval results as compliance evidence.


Arguments:
  file                       Path to Promptfoo JSON output file (from `promptfoo eval -o results.json`)


Options:
  --ai-system-id <id>        Link evidence to a registered AI system (UUID)
  --label <key=value>        Attach labels (repeatable, e.g., --label env=staging --label branch=main)
  --base-url <url>           TruthVouch API URL (default: from stored config)
  --json                     Output raw JSON response
  -h, --help                 Show help

API Reference

Endpoint

POST /api/v1/compliance/evidence/import/promptfoo

Request body

{
  "promptfooOutput": {
    "version": 3,
    "timestamp": "2026-04-01T10:30:00Z",
    "results": [
      {
        "provider": { "id": "openai:gpt-4o" },
        "prompt": { "raw": "What is the capital of France?" },
        "success": true,
        "score": 0.95,
        "gradingResult": {
          "pass": true,
          "score": 0.95,
          "namedScores": {
            "faithfulness": 0.98,
            "toxicity": 0.0
          }
        }
      }
    ],
    "stats": { "successes": 8, "failures": 2, "errors": 0 }
  },
  "aiSystemId": "optional-uuid-for-linking",
  "labels": {
    "environment": "staging",
    "branch": "feature/x"
  }
}

Response

{
  "totalResultsProcessed": 10,
  "evidenceCreated": 8,
  "evidenceSkipped": 2,
  "obligationLinksCreated": 5,
  "evidenceIds": [
    "550e8400-e29b-41d4-a716-446655440000",
    "550e8400-e29b-41d4-a716-446655440001"
  ],
  "warnings": [
    "Unknown assertion type 'custom:foo' skipped for obligation mapping"
  ]
}

HTTP status codes

  • 201 Created — One or more new evidence records created
  • 200 OK — All results skipped (already imported), no new records
  • 400 Bad Request — Invalid JSON, missing version, empty results, or > 500 results
  • 401 Unauthorized — Missing or invalid API key
  • 403 Forbidden — Insufficient tier (requires Professional+)

Example: cURL

Terminal window
PROMPTFOO_JSON=$(cat results.json)
curl -X POST https://api.truthvouch.com/api/v1/compliance/evidence/import/promptfoo \
  -H "Authorization: Bearer vt_live_your_api_key" \
  -H "Content-Type: application/json" \
  -d "{
    \"promptfooOutput\": $PROMPTFOO_JSON,
    \"aiSystemId\": \"optional-uuid\",
    \"labels\": {
      \"env\": \"production\"
    }
  }"

CI/CD Integration

GitHub Actions

Automatically import Promptfoo results to TruthVouch after each eval run:

name: LLM Eval + Compliance


on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * 1'  # Weekly Monday 6am


jobs:
  eval-and-import:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20


      - name: Install tools
        run: |
          npm install -g promptfoo @truthvouch/cli


      - name: Run Promptfoo eval
        run: promptfoo eval -c promptfooconfig.yaml -o results.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}


      - name: Import to TruthVouch
        run: |
          truthvouch login --api-key ${{ secrets.TRUTHVOUCH_API_KEY }}
          truthvouch import promptfoo results.json \
            --label env=ci \
            --label branch=${{ github.ref_name }}
        env:
          TRUTHVOUCH_API_URL: ${{ secrets.TRUTHVOUCH_API_URL }}


      - name: Upload results artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: promptfoo-results
          path: results.json
          retention-days: 30

GitLab CI

Run periodic evals and import results as part of your CI pipeline:

stages:
  - test


promptfoo-eval:
  stage: test
  image: node:20
  script:
    - npm install -g promptfoo @truthvouch/cli
    - promptfoo eval -c promptfooconfig.yaml -o results.json
    - truthvouch login --api-key $TRUTHVOUCH_API_KEY
    - truthvouch import promptfoo results.json --label env=ci --label branch=$CI_COMMIT_BRANCH
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY
    TRUTHVOUCH_API_URL: $TRUTHVOUCH_API_URL
  artifacts:
    paths:
      - results.json
    expire_in: 30 days
  only:
    - main
    - merge_requests

Evidence Mapping Details

Summary Evidence

For every unique (eval_timestamp, provider_id) pair in your Promptfoo JSON, TruthVouch creates a summary evidence record:

{
  "evidenceType": "promptfoo_eval",
  "title": "Promptfoo Eval — GPT-4o — 2026-04-01",
  "content": "Promptfoo eval completed: 8/10 tests passed (80.0%). Provider: openai:gpt-4o. 2 failures detected: 1 faithfulness, 1 harmful:hate.",
  "metadata": {
    "source": "promptfoo",
    "promptfooVersion": 3,
    "evalTimestamp": "2026-04-01T10:30:00Z",
    "providers": ["openai:gpt-4o"],
    "stats": { "successes": 8, "failures": 2, "errors": 0 },
    "tokenUsage": { "total": 2000 },
    "passRate": 0.80,
    "findingCategories": ["faithfulness", "harmful:hate"]
  }
}

Per-Failure Evidence

For each failing test result, TruthVouch creates a detailed evidence record for investigation:

{
  "evidenceType": "promptfoo_eval",
  "title": "Promptfoo Finding — harmful:hate — GPT-4o",
  "content": "Red-team test failed: harmful:hate assertion triggered. Provider: openai:gpt-4o. Score: 0.15. Reason: Output contained hate speech indicators.",
  "metadata": {
    "source": "promptfoo",
    "findingType": "red_team_failure",
    "assertionType": "harmful:hate",
    "provider": "openai:gpt-4o",
    "score": 0.15,
    "promptLabel": "adversarial-prompt-3",
    "gradingReason": "Output contained hate speech indicators"
  }
}

FAQ

What Promptfoo versions are supported?

Version 2 and version 3 JSON output formats are supported. Use promptfoo eval --version to check your version.

What’s the file size limit?

Maximum 10MB per request, covering up to ~500 test cases. For larger eval runs, split into multiple files or requests.

What happens when I re-import?

Re-importing the same file is safe — all records are skipped (zero new evidence). This is useful for periodic re-imports of the same eval file without manual deduplication.

But: If you run Promptfoo again with new results (updated models, new test cases, etc.), only the new results are imported. Idempotency is based on the eval timestamp and provider ID, so different evals always create new evidence.

Currently, one evidence record links to at most one AI system. To document eval results for multiple systems, run separate imports with different --ai-system-id values.

How are assertions mapped to obligations?

Assertion types (e.g., harmful:hate, pii:direct) are automatically mapped to compliance obligations using hardcoded framework hints. For custom assertion types or custom mappings, contact support or use the REST API directly with custom framework hints.

What if Promptfoo has no results?

Importing an empty results array will return a 400 Bad Request error. Ensure your eval config has at least one test case.

Does TruthVouch store my prompts or model outputs?

No. TruthVouch stores only the structured evaluation metadata (scores, assertions, provider IDs) and ignores sensitive data like raw prompts or model responses. Your compliance audit trail is maintained without exposing model behavior.

Troubleshooting

Authentication failed

Error: Authentication failed. Invalid or expired API key.
Suggestion: Run `truthvouch login` to re-authenticate.

Solution: Ensure your API key is valid and hasn’t expired. Generate a new key from Settings > API Keys in the TruthVouch dashboard.

File not found

Error: File not found: /path/to/results.json
Suggestion: Ensure the path is correct and the file exists.

Solution: Check that the file path is correct and the file exists. Use promptfoo eval -o results.json to generate it first.

Invalid Promptfoo JSON

Error: Invalid Promptfoo JSON structure
Suggestion: Expected format with "version" and "results" fields.

Solution: Ensure you’re using promptfoo eval -o results.json to generate the output. Manually edited JSON may be malformed.

Rate limit exceeded

Error: Rate limit exceeded.
Suggestion: Please wait and try again.

Solution: Wait a few minutes and retry. Each import counts as one API request. Professional tier: 5,000 requests/hour. Enterprise tier: 10,000 requests/hour.

Insufficient permissions

Error: Insufficient permissions.
Suggestion: Your API key may not have the required scope. Check your subscription tier.

Solution: Upgrade to Professional tier to use the Promptfoo import feature.

Next Steps

  • Monitor trends: Set up a weekly eval schedule and track your compliance posture over time
  • Auto-link obligations: Use the --ai-system-id flag to automatically link evidence to your AI inventory
  • Custom mappings: Contact support to configure custom assertion-to-obligation mappings for your organization
  • Webhook receiver: Interested in real-time webhook support for Promptfoo results? Contact us