promptfoo

Executes structured test suites defined in YAML/JSON config files against LLM prompts, agents, and RAG systems. The evaluator engine (src/evaluator.ts) parses test configurations containing prompts, variables, assertions, and expected outputs, then orchestrates parallel execution across multiple test cases with result aggregation and reporting. Supports dynamic variable substitution, conditional assertions, and multi-step test chains.

Executes identical test suites against multiple LLM providers (OpenAI, Anthropic, Google, AWS Bedrock, Ollama, etc.) and generates side-by-side comparison reports. The provider system (src/providers/) implements a unified interface with provider-specific adapters that handle authentication, request formatting, and response normalization. Results are aggregated with metrics like latency, cost, and quality scores to enable direct model comparison.

Supports streaming responses from LLM providers and enables token-level evaluation via callbacks that process partial responses as they arrive. The provider system handles streaming protocol differences (Server-Sent Events for OpenAI, event streams for Anthropic) and normalizes them into a unified callback interface. Enables measuring time-to-first-token, streaming latency, and token-level quality metrics.

Supports parameterized prompts with variable substitution, conditional blocks, and computed values. The prompt processor (Utilities and Output Generation in DeepWiki) parses template syntax (e.g., `{{variable}}`, `{{#if condition}}...{{/if}}`) and substitutes values from test case inputs or computed expressions. Enables testing prompt variations without duplicating test cases.

Validates LLM outputs against JSON schemas and grades structured outputs (JSON, YAML) for format compliance and content correctness. The assertion system supports JSON schema validation (via ajv library) and enables grading both schema compliance and semantic content. Supports extracting values from structured outputs for further evaluation.

Estimates API costs for evaluation runs by tracking token usage (input/output tokens) and applying provider-specific pricing. The evaluator aggregates token counts across test cases and providers, then multiplies by current pricing to estimate total cost. Supports both fixed pricing (per-token) and dynamic pricing (e.g., cached tokens in Claude). Enables cost-aware evaluation planning.

Generates adversarial test cases and attack prompts to identify security, safety, and alignment vulnerabilities in LLM applications. The red team system (Red Team Architecture in DeepWiki) uses a plugin-based attack strategy framework with built-in strategies (jailbreak, prompt injection, PII extraction, etc.) and integrates with attack providers that generate targeted adversarial inputs. Results are graded against safety criteria to identify failure modes.

Evaluates LLM outputs against multiple assertion types (exact match, regex, similarity, custom functions, LLM-based graders) and computes aggregated quality metrics. The assertions system (Assertions and Grading in DeepWiki) supports deterministic checks (string matching, JSON schema validation) and probabilistic graders (semantic similarity, LLM-as-judge). Results are scored and aggregated to produce pass/fail verdicts and quality percentages per test case.

Integrates LLM evaluation into continuous integration workflows via CLI commands, GitHub Actions, and exit code-based test gating. The CLI system (CLI Architecture in DeepWiki) provides `promptfoo eval` command that runs test suites and returns exit codes indicating pass/fail status. Results can be compared against baseline metrics to gate deployments; integration with version control enables tracking evaluation history per commit.

Provides a local web UI (Web Interface in DeepWiki) for exploring evaluation results with interactive filtering, search, and side-by-side comparison views. The frontend (React-based state management) loads test results and enables filtering by provider, assertion type, or test case; the backend server (Backend Server in DeepWiki) serves results and handles real-time updates. Results can be shared via shareable URLs or self-hosted deployments.

Supports evaluating custom or self-hosted LLM models via HTTP provider abstraction that accepts arbitrary OpenAI-compatible or custom API endpoints. The HTTP provider (HTTP Provider in DeepWiki) handles request/response transformation, enabling integration of models not natively supported by promptfoo (e.g., local Ollama instances, private fine-tuned models, or proprietary APIs). Supports custom request/response mapping via configuration.

Executes Python scripts or shell commands as LLM providers, enabling integration of custom models, local inference engines, or complex evaluation pipelines. The Python/Script providers (Python and Script Providers in DeepWiki) spawn subprocesses that receive test inputs via stdin/arguments and return outputs via stdout. Supports arbitrary custom logic without requiring native API integration.

Stores evaluation results in a local database (SQLite by default) and enables comparing current test runs against historical baselines to detect quality regressions. The data models and persistence layer (Data Models and Persistence in DeepWiki) serialize test results with metadata (timestamp, provider, config hash) enabling trend analysis. Supports querying results by date range, provider, or test case to identify when quality degraded.

Integrates AWS Bedrock models (Claude, Llama, Mistral, etc.) via unified provider interface with automatic credential handling via AWS SDK. The Bedrock provider (AWS Bedrock Integration in DeepWiki) handles model invocation, streaming, and response parsing. Supports both on-demand and provisioned throughput models with cost tracking. Extends to other cloud providers (Google Vertex AI, Azure OpenAI) via similar adapter patterns.