dagster

Enables developers to define data assets as Python functions decorated with @asset, automatically constructing a directed acyclic graph (DAG) of dependencies through function parameter matching and explicit asset_deps declarations. The system parses asset definitions at load time, resolves dependencies via asset keys, and builds an in-memory graph representation that tracks lineage, partitioning schemes, and materialization requirements without requiring manual DAG specification.

Implements a sophisticated partitioning system allowing assets to be divided across time-based (daily, hourly), static categorical, or dynamically-generated partitions, with support for multi-dimensional partitioning (e.g., date × region). The system tracks partition state, enables targeted backfills, and optimizes execution by only materializing changed partitions. Partition definitions are composable and integrate with the asset graph to automatically determine which partitions need execution.

Tracks asset freshness (time since last materialization) and health status (latest run success/failure) via the asset health system. Freshness policies define expected materialization intervals (e.g., daily); the system compares actual freshness against policies and marks assets as stale. Health status is queryable via GraphQL and can trigger alerts via sensors. Integration with external systems (Slack, PagerDuty) enables notifications when assets become unhealthy.

Provides AssetSelection API enabling programmatic selection of assets based on keys, tags, groups, or custom predicates. Selections can be composed (union, intersection, difference) and used to target specific assets for execution, backfills, or queries. The system resolves dependencies automatically, ensuring upstream assets are included in execution. Selections are queryable via GraphQL, enabling external systems to discover which assets will be executed.

Implements a configuration system enabling assets, resources, and jobs to accept configuration dictionaries at definition or execution time. Configuration is specified via ConfigurableResource base class or @resource decorator, with schema validation via Pydantic. Environment-specific configs are loaded from YAML files or environment variables, enabling dev/staging/prod deployments without code changes. Configuration is resolved at execution time and injected into asset context.

Tracks asset versions based on code changes, enabling detection of when asset definitions change and triggering re-materialization of downstream assets. Asset lineage is reconstructed from event logs, showing data flow across the pipeline. Data contracts (input/output schemas) can be defined on assets, with validation at execution time to detect schema mismatches. Lineage is queryable via GraphQL and visualizable in the UI.

Captures detailed execution events (AssetMaterializationEvent, DagsterEventType) during asset computation, including execution time, data quality metrics, row counts, and custom metadata. Events are persisted to configurable event log storage (SQLite, PostgreSQL, in-memory) and queryable via GraphQL, enabling real-time monitoring, data lineage reconstruction, and post-execution analysis without requiring external observability tools.

Provides two complementary automation mechanisms: Sensors poll external systems (databases, APIs, file systems) on a configurable interval to detect changes and trigger asset materialization, while Schedules execute assets on cron expressions or custom timing logic. Both are defined as Python functions decorated with @sensor or @schedule, integrated into the asset daemon that runs continuously to evaluate automation rules and submit runs to the executor.

Implements a dependency injection system where assets and ops declare required resources (databases, APIs, cloud storage) as function parameters, resolved at execution time from a configured resource dictionary. I/O managers abstract data persistence, enabling assets to read/write data to configurable backends (filesystem, S3, Snowflake, BigQuery) without hardcoding storage logic. Resources are scoped (process, step) and support initialization/cleanup hooks, enabling connection pooling and resource lifecycle management.

Exposes a comprehensive GraphQL API (dagster_graphql package) enabling queries on asset definitions, run history, event logs, and partition state. Queries execute against DagsterInstance storage, supporting filtering by asset key, run status, time range, and partition. The API includes mutations for triggering runs, launching backfills, and managing dynamic partitions. Workspace context provides multi-tenant isolation and permission scoping, enabling role-based access control in cloud deployments.

Integrates dbt projects as Dagster assets via the dagster-dbt library, automatically loading dbt models as asset definitions and tracking their dependencies. The integration captures dbt metadata (column descriptions, tests, freshness) and syncs it into Dagster's asset system. Supports dbt cloud execution via API or local dbt CLI invocation, with event parsing to capture dbt test results and model execution metrics as Dagster events.

Provides the Pipes framework (dagster-pipes package) enabling Dagster to orchestrate external processes (Spark jobs, Kubernetes pods, Lambda functions) while capturing their output as Dagster events. External processes write events to stdout in a structured format, which Dagster parses and converts to AssetMaterializationEvent, DagsterEventType, and custom events. Supports multiple execution contexts (Kubernetes, ECS, Spark) with language-agnostic client libraries (Python, Java, Go).

Implements a backfill system enabling targeted re-execution of asset partitions across time ranges or custom selections. Backfills are submitted as BackfillRequest objects specifying asset selection and partition range, then executed by the executor with automatic dependency resolution. The system tracks backfill progress, enables cancellation, and optimizes execution by only materializing partitions that don't already exist or have changed upstream.

Provides pluggable executors (in-process, multiprocess, Kubernetes, Celery) that determine how ops and assets are executed. The multiprocess executor spawns worker processes for parallel execution with configurable concurrency limits and resource tags. Kubernetes executor submits jobs as Kubernetes pods with resource requests/limits. Executors integrate with the run launcher to manage process lifecycle, capture output, and handle failures with configurable retry logic.