Evidently AI

Detects distribution shifts in production data by computing statistical tests (Kolmogorov-Smirnov, chi-square, Jensen-Shannon divergence) across numerical and categorical columns. Evidently's drift detection engine compares reference datasets against production batches using a modular metric system that abstracts statistical computation into pluggable test implementations, enabling both univariate and multivariate drift signals with configurable thresholds and preset bundles (DataDriftPreset) for rapid deployment.

Executes pass/fail validation on model performance metrics (accuracy, precision, recall, F1, ROC-AUC) by composing TestSuite objects with condition-based assertions. The framework evaluates predictions against ground truth labels using a test condition system that supports threshold comparisons, relative change detection, and statistical significance tests. Results integrate directly into CI/CD pipelines via JSON export and CLI commands, enabling automated regression detection without manual threshold tuning.

Extracts row-level text features (sentiment, toxicity, readability, length, language) using a descriptor system where each Descriptor subclass implements a specific feature extraction logic. Descriptors are applied to text columns to generate new columns, which are then aggregated into batch-level metrics. The framework supports both built-in descriptors (using heuristics or lightweight models) and custom descriptors (using external NLP models or APIs).

Enables automated validation in CI/CD pipelines by executing TestSuite objects that return pass/fail results and exit codes. Test suites can be triggered via CLI commands, returning non-zero exit codes on failure to halt deployment. Results are exported as JSON for integration with CI/CD platforms (GitHub Actions, GitLab CI, Jenkins), enabling automated quality gates without custom scripting.

Enables evaluation of metrics within subpopulations by specifying group columns in ColumnMapping, allowing segment-level analysis without manual data filtering. Metrics are computed separately for each group, enabling detection of performance disparities across demographic segments, geographic regions, or other categorical dimensions. Results are aggregated and visualized with group-level breakdowns.

Evaluates large language model outputs using a descriptor-based architecture that extracts text features (sentiment, toxicity, readability, answer relevance) and computes statistical aggregations across batches. Descriptors are row-level feature extractors that apply NLP models or heuristics to generate columns, which are then aggregated into batch-level metrics. The framework supports both reference-based metrics (comparing LLM output to ground truth) and reference-free metrics (assessing output properties directly), with integration to external LLM APIs for semantic evaluation.

Generates web-based dashboards that visualize metrics and test results with interactive filtering, time-series plots, and drill-down capabilities. The dashboard system consumes metric snapshots from reports and test suites, stores them in a backend (file-based or cloud), and renders them via a React-based UI. Real-time monitoring is enabled through a collection API that accepts metric batches, persists them to storage, and updates dashboard views without requiring full report recomputation.

Enables extension of Evidently's metric system by subclassing Metric and TestCondition base classes, allowing users to implement domain-specific evaluations without modifying framework code. Custom metrics integrate into the unified PythonEngine execution model, enabling composition with built-in metrics in reports and test suites. The plugin architecture supports custom descriptors for text analysis, custom statistical tests, and custom aggregation logic.

Computes comprehensive data quality metrics across tabular datasets using a preset system that bundles related metrics (DataQualityPreset, DataDriftPreset, TextEvals). Presets encapsulate metric selection, threshold defaults, and visualization templates, enabling one-line data profiling without manual metric composition. The framework supports both reference-based comparison (comparing production to reference dataset) and standalone profiling (assessing data properties directly).

Executes a collection of metrics through the PythonEngine and serializes results into multiple formats: interactive HTML reports with visualizations, JSON exports for programmatic consumption, and Python Snapshot objects for in-memory analysis. The Report class orchestrates metric computation, result aggregation, and format rendering without requiring users to manage execution details. Results are cached as Snapshot objects, enabling efficient re-rendering and downstream processing.

Automatically infers column types (numerical, categorical, text, datetime) from Pandas DataFrames and generates ColumnMapping objects that specify feature roles (target, prediction, reference, group). The mapping system enables the framework to apply appropriate metrics and statistical tests without manual configuration. Users can override inferred types and define custom column roles for specialized evaluation scenarios.

Tracks metrics across multiple evaluation runs (e.g., daily monitoring batches) and enables historical comparison via the monitoring dashboard and collection API. The framework stores metric snapshots with timestamps, enabling visualization of metric trends, detection of gradual degradation, and comparison against historical baselines. Test conditions can reference historical statistics (e.g., mean, percentile) for adaptive thresholding.

Implements a suite of statistical tests (Kolmogorov-Smirnov, chi-square, Jensen-Shannon divergence, t-test, Mann-Whitney U) that can be applied to detect significant differences between distributions or groups. Tests are encapsulated as metric classes that compute p-values and test statistics, enabling integration into reports and test suites. Users can configure test selection, significance levels, and multiple comparison corrections.