Soda

Parses human-readable SodaCL check definitions into an abstract syntax tree (AST) that is then compiled into executable check objects. The SodaCL parser (sodacl_parser.py) tokenizes and validates check syntax, supporting metric thresholds, distribution checks, anomaly detection rules, and freshness conditions. This compilation step decouples check definition from execution, enabling the same checks to run against multiple data sources without modification.

Converts compiled SodaCL checks into dialect-specific SQL queries for execution against the target data source. The Query Execution System (referenced in architecture) generates optimized SQL for PostgreSQL, Snowflake, BigQuery, Redshift, Spark, Athena, and Spark DataFrames, handling dialect differences (e.g., window functions, date arithmetic, NULL handling). Each data source package (soda-core-postgres, soda-core-snowflake, etc.) provides a QueryBuilder that translates abstract check definitions into native SQL.

Provides command-line interface for executing scans ('soda scan'), testing data source connections ('soda test-connection'), updating distribution reference files ('soda update-dro'), and ingesting dbt results ('soda ingest'). The CLI parses command-line arguments, loads configuration, and delegates to the Scan orchestrator. Supports output formatting (JSON, YAML) and variable substitution via command-line flags.

Monitors table schemas for unexpected changes (added/removed/renamed columns, type changes) by comparing current schema against a baseline. Enables checks like 'schema(missing_columns: [id, name])' to ensure required columns exist. The schema validation is performed as part of the check execution, comparing actual table structure against expected structure defined in checks.

Evaluates computed metrics (row count, missing values, duplicates, etc.) against user-defined thresholds using comparison operators (>, <, ==, >=, <=, between). The Metric Checks component executes a SQL query to compute the metric, then applies the threshold logic to determine pass/fail status. Supports both absolute values and percentage-based thresholds, enabling checks like 'missing_count(email) < 5' or 'invalid_percent(phone) <= 2%'.

Captures the statistical distribution of a column (via 'soda update-dro' CLI command) and stores it as a Distribution Reference Object (DRO) file. On subsequent scans, compares the current column distribution against the stored reference using statistical tests to detect anomalies. The Scientific package integrates Prophet time-series forecasting for advanced anomaly detection, identifying unexpected shifts in data patterns beyond simple threshold violations.

Profiles columns to compute statistics (min, max, mean, median, stddev, cardinality, missing count) and samples rows that fail quality checks for root cause analysis. When a check fails, Soda can optionally retrieve and store a sample of the failing rows (up to a configurable limit) along with their column values, enabling data engineers to investigate data quality issues without querying the warehouse manually.

Monitors data freshness by comparing the maximum timestamp in a column (e.g., max(updated_at)) against the current time, ensuring data is updated within a specified time window (e.g., 'updated_at < 1 hour ago'). Supports both absolute time windows and relative thresholds, enabling checks like 'freshness(created_at) < 24h' that automatically adapt to the current time.

The Scan class (scan.py) orchestrates the entire check execution lifecycle: loading configuration, connecting to data sources, parsing SodaCL checks, executing queries, evaluating results, and generating reports. Manages state across multiple checks, handles errors gracefully, and coordinates integration with external systems (Soda Cloud, dbt). The Scan object is the primary entry point for programmatic use of Soda Core.

Integrates with dbt by ingesting dbt test results and converting them into Soda checks for centralized monitoring. The dbt_config.py and dbt.py modules enable Soda to read dbt test outputs and correlate them with dbt metadata (lineage, documentation). Supports the 'soda ingest' CLI command to import dbt test results into Soda Cloud for unified data quality visibility.

Loads and parses YAML configuration files (checks.yml, data_sources.yml) with support for variable substitution, environment variables, and parameterized checks. The configuration_parser.py module validates configuration syntax, resolves variable references (e.g., ${ENV_VAR}), and builds in-memory configuration objects. Enables environment-specific configurations (dev, staging, prod) without duplicating check definitions.

Integrates with Soda Cloud (SaaS platform) to send scan results, failed row samples, and check metadata to a centralized dashboard. Enables cross-warehouse monitoring, alerting, and incident tracking without running a separate monitoring infrastructure. The integration is optional; Soda Core can run standalone without Cloud connectivity.