Dagster

Defines data assets as Python functions decorated with @asset, automatically inferring upstream/downstream dependencies through function parameters and return type annotations. The asset system builds a directed acyclic graph (DAG) at definition time, enabling Dagster to understand the full data lineage without explicit edge declarations. Assets are versioned, partitionable, and support multi-output patterns through Out() objects, creating a type-safe, code-first alternative to YAML-based DAG definitions.

Implements a type-aware I/O abstraction layer where each asset's input/output is validated against declared types before and after execution. I/O managers (implementations of IOManager interface) handle serialization, deserialization, and storage location logic, decoupling asset code from storage details. Dagster provides built-in managers for Pandas DataFrames, Polars, Parquet, and cloud storage (S3, GCS, ADLS); custom managers can be registered per asset or globally, enabling seamless switching between local development (in-memory) and production (cloud storage) without code changes.

Dagster+ is a managed cloud service that hosts Dagster instances with automatic scaling, monitoring, and multi-workspace support. Code locations are Git repositories containing Definitions objects that are deployed to Dagster+ via the dg CLI or GitHub integration. Dagster+ automatically pulls code from Git, installs dependencies, and deploys code locations without manual infrastructure management. Supports multiple code locations per workspace, enabling teams to deploy assets from different repositories independently. Includes built-in secret management, audit logging, and RBAC (role-based access control). Integrates with cloud executors (Kubernetes, ECS) for distributed execution.

Provides a lightweight framework for executing external processes (Python scripts, shell commands, Spark jobs) from Dagster assets while maintaining type safety and data passing. The Pipes framework uses a message-passing protocol over stdout/stderr to communicate between the parent Dagster process and child processes. Child processes emit structured messages (logs, metrics, asset materializations) that are captured and stored in the event log. Supports arbitrary data passing via context.log_event() in child processes. Eliminates the need for intermediate files or databases for inter-process communication.

Enables assets/ops to emit multiple outputs dynamically at runtime using DynamicOutput objects. Each output is tagged with a unique key, creating multiple downstream assets/ops that process each output independently. Supports fan-out (one asset produces multiple outputs) and fan-in (multiple outputs are collected into a single downstream asset). Dynamic outputs are useful for conditional branching (e.g., process different data based on a condition) and parallel processing of variable-length lists. Downstream assets can be defined to consume all dynamic outputs or specific subsets via output filtering.

Tracks asset versions based on code changes and upstream dependencies. Each asset materialization is tagged with a version identifier that captures the asset's code hash and upstream asset versions. Enables querying historical versions of assets and re-materializing specific versions without code changes. Version lineage is tracked in the event log, enabling time-travel queries (e.g., 'get asset X as it was on 2024-01-01'). Supports version-aware I/O managers that store multiple versions of the same asset. Useful for debugging (reproduce results from a specific version) and compliance (audit trail of data transformations).

Provides two complementary automation mechanisms: Schedules execute assets on fixed time intervals (cron-like), while Sensors poll external systems (databases, APIs, S3 buckets) for state changes and trigger asset runs conditionally. Both are defined as Python functions decorated with @schedule or @sensor, returning RunRequest objects that specify which assets to materialize. The Asset Daemon (a long-running process) executes tick logic at intervals, evaluating sensor conditions and schedule times, then submitting runs to the executor. Supports dynamic partitioning where sensor logic can emit multiple RunRequests with different partition keys in a single tick.

Enables assets to be partitioned by time (daily, hourly, monthly), discrete values (regions, customers), or dynamic ranges computed at runtime. Partitioning is declared via @asset(partitions_def=...) and automatically generates partition keys. The system tracks which partitions have been materialized, enabling incremental runs that only process new/missing partitions. Backfill operations can target specific partition ranges or use dynamic partition discovery (e.g., query a database to find new customer IDs). Partition dependencies are resolved automatically — if asset B depends on asset A and both are partitioned, Dagster ensures partition B_1 only runs after A_1 completes.

Captures structured events during asset execution (start, success, failure, logs, metrics) and stores them in an event log database (SQLite, PostgreSQL, or cloud-native stores). Each run generates a stream of DagsterEvent objects that are persisted and queryable via GraphQL. Asset health is computed from recent materialization history — Dagster tracks freshness (time since last materialization), materialization frequency, and failure rates. The Dagster UI visualizes asset lineage, run history, and health status in real-time. Custom events can be emitted from asset code via context.log_event(), enabling domain-specific observability (e.g., row counts, data quality metrics).

Defines reusable, environment-specific configurations (database connections, API clients, cloud credentials) as Resource objects that are injected into asset/op code via the context parameter. Resources are registered in a Definitions object and can be overridden per deployment (dev, staging, prod) without code changes. The context object provides access to resources, logging, run metadata, and partition keys. Resources support dependency injection — a resource can depend on other resources (e.g., a database resource depending on a credentials resource). This pattern eliminates hardcoded credentials and enables testing with mock resources.

Exposes a comprehensive GraphQL schema for querying pipeline state, asset metadata, run history, and event logs. The API is implemented in dagster-graphql module and serves as the backend for the Dagster UI. Queries support filtering by asset key, run status, partition, and time range. Mutations enable run submission, cancellation, and asset materialization requests. The schema includes types for AssetNode (asset metadata), Run (execution record), Event (structured log entry), and Partition (partition status). Subscriptions are not supported; polling is required for real-time updates.

Executes asset/op code using pluggable Executor implementations that determine how and where tasks run. Built-in executors include: InProcessExecutor (single-process, for testing), MultiprocessExecutor (spawns child processes for parallelism), and DagsterDaemonExecutor (submits runs to a daemon queue). Cloud executors (Kubernetes, ECS, Databricks) are provided via integration libraries. Executors receive a RunRequest and execute ops/assets in topological order, respecting dependencies. The executor is selected per job/asset and can be overridden at runtime via tags. Supports resource limits (max workers, memory) and custom executor implementations.

Automatically generates Dagster assets from dbt models, seeds, and snapshots via the @dbt_assets decorator. The integration parses dbt's manifest.json to extract model dependencies and creates corresponding Dagster assets with proper lineage. dbt models are materialized through a DbtCliResource that executes dbt commands (dbt run, dbt test). Lineage is automatically inferred from dbt's dependency graph — upstream dbt models become asset dependencies. Supports dbt tests as asset checks (validations that run after materialization). The integration handles dbt-specific features (macros, variables, selectors) transparently.

Defines data quality checks as Python functions decorated with @asset_check that validate asset outputs after materialization. Checks receive the asset's data and return a boolean or structured result (AssetCheckResult). Checks can be blocking (fail the asset if check fails) or non-blocking (log failure but continue). Multiple checks can be attached to a single asset. Check results are stored in the event log and visualized in the Dagster UI. Checks support custom metadata (description, tags) and can be filtered/queried via GraphQL. Integrates with dbt tests (dbt test results are converted to asset checks).