DeepEval

Executes evaluation metrics using any LLM provider (OpenAI, Anthropic, Ollama, local models) as a judge through a unified model abstraction layer. DeepEval abstracts provider-specific APIs into a common interface, routing metric prompts to the configured LLM and parsing structured outputs (scores, reasoning) via schema-based deserialization. Supports both synchronous and asynchronous evaluation with built-in retry logic and token counting for cost tracking.

Provides 50+ pre-built evaluation metrics including faithfulness, answer relevancy, contextual recall, hallucination detection, bias, toxicity, and RAG-specific metrics (retrieval precision, context utilization). Each metric inherits from a BaseMetric class defining the measure() interface and is implemented using LLM-as-judge prompts (G-Eval style), statistical methods (ROUGE, BERTScore), or specialized NLP models (toxicity classifiers). Metrics are composable and can be combined into evaluation suites.

Provides benchmark functionality to compare LLM model performance across evaluation datasets using standardized metrics. Benchmarks define a set of models, datasets, and metrics to evaluate, and produce comparison reports showing performance differences. Supports benchmarking against published datasets (MMLU, HellaSwag, etc.) and custom datasets. Results are tracked over time, enabling trend analysis and regression detection. Benchmark reports include statistical significance testing and visualization of performance differences.

Provides prompt optimization capabilities to iteratively improve LLM prompts based on evaluation metrics. Supports A/B testing of different prompt variants against the same evaluation dataset, measuring performance differences using metrics like answer relevancy and hallucination. Optimization strategies include prompt template variation, few-shot example selection, and instruction refinement. Results are tracked and compared, enabling data-driven prompt engineering. Optimized prompts can be versioned and deployed to production.

Manages test run lifecycle including execution, result storage, and historical tracking. Each test run captures metadata (timestamp, model version, dataset version, metrics evaluated, pass rate) and individual test results (metric scores, pass/fail status). Test runs are persisted locally (JSON/SQLite) or in Confident AI cloud backend, enabling historical comparison and regression detection. Supports filtering and querying test runs by date, model, dataset, or metric. Test run reports can be exported for analysis or shared with stakeholders.

Provides a unified Model abstraction layer (deepeval/models/base.py) that normalizes APIs across 10+ LLM providers (OpenAI, Anthropic, Ollama, vLLM, Azure, Bedrock, etc.). Each provider has a concrete implementation that translates DeepEval's generic model interface (generate(), generate_async()) to provider-specific APIs. Model configuration is centralized, supporting environment variables, config files, and programmatic initialization. Supports model-specific features (temperature, max_tokens, system prompts) while maintaining a consistent interface.

Provides command-line interface (CLI) for running evaluations, managing datasets, and configuring projects without writing Python code. CLI commands support test execution (deepeval test), dataset operations (deepeval dataset), and cloud integration (deepeval login). Configuration is managed through YAML files (deepeval.yaml) and environment variables, enabling reproducible evaluation workflows and CI/CD integration. CLI output includes human-readable result summaries and machine-readable JSON export for integration with external tools.

Integrates DeepEval metrics into pytest test discovery and execution via a pytest plugin (deepeval/plugins/pytest_plugin.py). Test cases are defined as pytest test functions decorated with @pytest.mark.deepeval, and metrics are asserted using standard pytest assertions. The plugin captures test results, manages test runs, and exports results to the Confident AI platform or local storage. Supports parallel test execution, test filtering, and integration with CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins).

Provides a dataset abstraction (EvaluationDataset class) for managing collections of test cases with version control, persistence, and synthetic data generation. Golden records are curated test cases stored in JSON/CSV format with input, expected output, and optional metadata. Datasets support CRUD operations, filtering, and export to multiple formats. Integrates with Confident AI platform for cloud-based dataset versioning and collaboration, enabling teams to maintain evaluation datasets across model iterations.

Provides distributed tracing capabilities via an @observe decorator that instruments LLM application code to capture execution spans (function calls, LLM invocations, tool calls). Spans form a hierarchical tree structure with parent-child relationships, enabling visualization of complex LLM workflows. Integrates with OpenTelemetry for standards-based tracing and exports spans to Confident AI dashboard or external observability platforms. Captures latency, token usage, errors, and custom attributes per span.

Allows developers to define custom metrics by subclassing BaseMetric and implementing a measure() method that accepts an LLMTestCase and returns a MetricResult. Custom metrics can use any evaluation logic (LLM-as-judge, statistical, ML models) and are validated against a schema defining required inputs (input, actual_output, expected_output, retrieval_context). The framework provides template prompts and helper functions for common patterns (LLM-as-judge via G-Eval, reference-based scoring). Custom metrics integrate seamlessly with the evaluation pipeline and can be combined with built-in metrics.

Implements a caching layer (deepeval/cache.py) that stores metric evaluation results keyed by test case hash and metric configuration, avoiding redundant evaluations of identical inputs. Cache is stored locally (SQLite) or in Confident AI cloud backend. Supports cache invalidation by metric version, test case modification, or explicit clearing. Caching is transparent to users — metrics check cache before execution and store results after completion.

Provides a ConversationSimulator that generates multi-turn dialogue datasets by simulating conversations between user and assistant LLMs. The simulator takes a conversation template (initial prompt, turn count, evaluation criteria) and generates realistic dialogue sequences. Supports different conversation styles (question-answering, task-oriented, open-ended) and can evaluate conversation quality using metrics like turn relevancy and coherence. Generated conversations are stored as ConversationalTestCase objects compatible with the evaluation pipeline.

Provides red teaming capabilities to generate adversarial test cases designed to expose weaknesses in LLM applications. Red teaming strategies include prompt injection, jailbreak attempts, edge case generation, and bias probing. The framework uses LLM-as-judge to generate adversarial inputs and evaluates system robustness using safety metrics (toxicity, bias, hallucination). Red teaming results are tracked separately from standard evaluation and can be used to identify failure modes and improve system resilience.

Provides guardrails (deepeval/guardrails.py) that validate and filter LLM outputs against user-defined rules before they reach end users. Guardrails can enforce constraints like output length, content filtering (toxicity, PII), format validation (JSON schema, regex), and custom business logic. Guardrails are composable and can be chained together. When a guardrail violation is detected, the system can reject the output, retry with a modified prompt, or flag for human review. Guardrails integrate with the evaluation pipeline to measure compliance.