sql transformation compilation and execution
Compiles SQL transformation code and executes it against connected data warehouses. Handles SQL parsing, optimization, and execution across multiple SQL dialects with native support for Snowflake, BigQuery, and Redshift.
dependency graph resolution and dag management
Automatically detects dependencies between data transformations and builds a directed acyclic graph (DAG) to determine execution order. Optimizes the dependency chain for efficient parallel execution.
workspace and environment management
Manages development, staging, and production environments with separate configurations and data warehouse schemas. Enables safe testing before production deployment.
performance profiling and optimization recommendations
Analyzes transformation execution performance, identifies bottlenecks, and provides optimization recommendations. Tracks execution metrics and suggests query improvements.
version control integration and change tracking
Integrates with Git and version control systems to track changes to transformations. Enables collaboration, code review, and rollback capabilities.
documentation generation and metadata publishing
Automatically generates documentation for data models, transformations, and lineage. Publishes metadata to data catalogs and documentation sites.
data quality testing and validation
Runs built-in data quality tests and schema validation on transformations to catch data issues early. Includes assertions for null checks, uniqueness, referential integrity, and custom validation rules without requiring external testing frameworks.
schema inference and management
Automatically infers and manages data schemas for transformations, detecting column types and structure changes. Validates schema consistency across the pipeline and alerts on breaking changes.
+6 more capabilities