DeepEval

Executes evaluation metrics by prompting LLMs (OpenAI, Anthropic, Ollama, etc.) to score LLM outputs against structured rubrics. Uses a metric execution pipeline that abstracts provider differences through a unified Model interface, enabling researchers to swap judge models without changing evaluation code. Supports both deterministic scoring (0-1 scale) and reasoning-based judgments via G-Eval and custom metric implementations.

Provides pre-built metric implementations covering RAG evaluation (faithfulness, answer relevancy, contextual recall), hallucination detection, bias/toxicity analysis, and conversation quality metrics. Each metric is implemented as a class inheriting from BaseMetric, with configurable thresholds, LLM judge selection, and custom scoring logic. Metrics can run in isolation or as part of a test suite, with caching to avoid redundant evaluations.

Integrates with the Confident AI cloud platform for centralized evaluation result storage, visualization, and team collaboration. Automatically syncs evaluation runs, metrics, and traces to the platform, enabling web-based dashboards for result exploration, trend analysis, and team sharing. Supports API-based access to evaluation history and results.

Provides tools for systematically testing and optimizing LLM prompts by running evaluations across multiple prompt variants and comparing metric scores. Supports A/B testing, multi-variant testing, and automated prompt generation. Integrates with the evaluation pipeline to track prompt performance and identify optimal prompts.

Abstracts LLM provider differences through a unified Model interface that handles provider-specific API calls, response parsing, and error handling. Supports OpenAI, Anthropic, Ollama, Azure OpenAI, and custom providers. Configuration is centralized and can be set via environment variables, config files, or programmatic API, enabling easy provider switching without code changes.

Provides pre-built benchmark suites (e.g., RAGAS, MTBE) that evaluate LLM systems against standardized datasets and metrics. Enables comparison of system performance against published benchmarks and other implementations. Supports custom benchmark definition and execution.

Provides command-line interface (CLI) for running evaluations, managing datasets, and configuring projects without writing Python code. CLI commands support test execution (deepeval test), dataset operations (deepeval dataset), and cloud integration (deepeval login). Configuration is managed through YAML files (deepeval.yaml) and environment variables, enabling reproducible evaluation workflows and CI/CD integration. CLI output includes human-readable result summaries and machine-readable JSON export for integration with external tools.

Integrates DeepEval metrics into pytest test discovery and execution via a custom pytest plugin. Test cases are defined as pytest test functions decorated with @pytest.mark.deepeval, executed through pytest's standard runner, and reported in JUnit XML format compatible with GitHub Actions, GitLab CI, and other CI/CD platforms. Supports parallel test execution, test filtering, and result aggregation.

Provides a structured dataset abstraction (EvaluationDataset) for managing collections of test cases with versioning, persistence, and CRUD operations. Golden records are canonical test cases stored in JSON or CSV format, versioned via git or the Confident AI platform, and loaded into memory for evaluation runs. Supports dataset splitting (train/test), filtering, and synthetic data generation via LLM prompting.

Implements distributed tracing via the @observe decorator, which wraps function calls and creates spans (trace records) with metadata (latency, inputs, outputs, errors). Spans form a hierarchical tree representing the call stack, enabling visibility into component-level behavior within complex LLM systems. Integrates with OpenTelemetry for export to observability platforms (Datadog, New Relic, etc.) and the Confident AI dashboard for visualization.

Implements a caching layer that stores metric evaluation results (score, reasoning, metadata) keyed by test case hash and metric configuration. Avoids redundant LLM API calls when the same test case is evaluated multiple times with the same metric. Cache is stored locally (in-memory or disk) or synced with the Confident AI platform for cross-machine consistency.

Enables developers to define custom metrics by subclassing BaseMetric and implementing a measure() method. Provides the G-Eval pattern (chain-of-thought reasoning with structured output) as a reusable template for building LLM-based metrics. Custom metrics can use any LLM provider, define custom scoring logic, and integrate with the evaluation pipeline without modifying core framework code.

Provides ConversationSimulator for generating multi-turn conversations between an LLM and a simulated user, enabling evaluation of conversational systems. Supports conversation history tracking, turn-level metrics (e.g., TurnRelevancy), and conversation-level metrics (e.g., coherence across turns). Enables testing of dialogue quality, context retention, and conversation flow without manual conversation data.

Provides tools for generating adversarial test cases and evaluating LLM robustness against attacks (prompt injection, jailbreaks, edge cases). Uses LLM-based generation to create adversarial inputs and metrics to measure system resilience. Integrates with the evaluation pipeline to track adversarial performance separately from standard metrics.

Provides a guardrails system for defining and enforcing safety constraints on LLM outputs (e.g., no toxic content, no PII leakage, no harmful instructions). Guardrails are evaluated as metrics and can block or flag outputs that violate constraints. Integrates with the evaluation pipeline to track constraint violations and generate safety reports.