deepeval

Executes evaluation metrics using LLMs as judges by constructing structured prompts with evaluation schemas and routing them to any LLM provider (OpenAI, Anthropic, Ollama, etc.). Implements the G-Eval pattern with research-backed scoring templates that normalize outputs to 0-1 scales. The metric execution pipeline handles provider abstraction, caching of LLM responses, and deterministic scoring through configurable model selection and temperature control.

Provides 50+ pre-built metrics covering general LLM quality (relevance, coherence, faithfulness), RAG-specific concerns (retrieval precision, context relevance), and conversation quality (turn-level relevance, conversation coherence). Each metric is implemented as a subclass of the Metric base class with built-in scoring logic that can use LLM-as-judge, statistical methods, or local NLP models. Metrics are composable and can be mixed in test runs to evaluate multiple dimensions simultaneously.

Provides guardrail metrics to evaluate safety and compliance of LLM outputs, including toxicity detection, PII redaction, prompt injection detection, and bias assessment. Guardrails can be applied as pre-generation filters or post-generation validators. Integrates with external safety APIs (e.g., OpenAI Moderation) and local NLP models for offline evaluation.

Generates adversarial test cases designed to expose weaknesses in LLM applications through systematic perturbation of inputs (e.g., typos, paraphrasing, edge cases). Red teaming metrics evaluate robustness by measuring how outputs change under adversarial conditions. Supports both automated generation and manual specification of adversarial scenarios.

Provides utilities for systematic prompt optimization by running evaluations across multiple prompt variants and comparing results. Supports A/B testing of prompts, model versions, and hyperparameters. Results are aggregated and compared to identify the best-performing variant. Integrates with the Confident AI platform for historical tracking of prompt iterations.

Provides a command-line interface (deepeval CLI) for running evaluations, managing datasets, and configuring projects. Supports configuration files (deepeval.json) for project settings, environment variables for API keys, and provider configuration management. CLI commands enable running evaluations without writing Python code, making it accessible to non-developers.

Defines evaluation test cases as structured Python dataclasses (LLMTestCase, ConversationalTestCase) that capture input, expected output, actual output, and context. The framework provides schema validation, serialization to JSON/CSV, and dataset-level operations (filtering, splitting, versioning). Test cases can be created manually, loaded from files, or generated synthetically using LLM-based data generation.

Orchestrates the execution of test cases against metrics using the evaluate() function, which handles parallel metric execution, result aggregation, and test run persistence. The execution engine manages metric scheduling, error handling, and result caching. Test runs are tracked with metadata (timestamp, model version, dataset version) and can be compared across iterations to detect regressions.

Generates synthetic test cases by prompting an LLM to create realistic input-output pairs based on seed data or templates. The synthesis engine uses configurable prompts to control the diversity and quality of generated cases. Generated cases are validated against the test case schema and can be filtered or augmented before being added to evaluation datasets.

Allows developers to define custom metrics by subclassing the Metric or GEval base class and implementing a measure() method. Custom metrics can use LLM-as-judge, statistical methods, or external APIs for scoring. The framework provides utilities for prompt templating, response parsing, and score normalization. Custom metrics integrate seamlessly with the evaluation pipeline and can be composed with built-in metrics.

Provides the @observe decorator to instrument individual functions within an LLM application, capturing inputs, outputs, and execution metadata as spans in a trace hierarchy. Traces are collected by the TraceManager and can be exported to OpenTelemetry or persisted to the Confident AI platform. Enables visibility into which components contribute to evaluation failures and supports production monitoring of LLM systems.

Integrates with pytest to allow evaluation metrics to be run as test assertions using the @test_case decorator. Test cases are discovered and executed by pytest, enabling LLM evaluations to be part of the standard testing workflow. Supports pytest fixtures, parametrization, and reporting. Failed evaluations are reported as test failures with detailed metrics output.

Supports evaluation of multi-turn conversations through the ConversationalTestCase data structure, which captures conversation history with turn-level metadata. Metrics can be evaluated at the conversation level (overall coherence) or turn level (individual response quality). The conversation simulator can generate synthetic multi-turn conversations for testing dialogue systems.

Integrates with the Confident AI platform to persist test runs, compare results across iterations, and track evaluation metrics over time. Test runs are uploaded with full metadata (model version, dataset version, timestamp) and can be queried via the platform dashboard. Enables regression detection and historical analysis of evaluation trends.