Apache Airflow

Enables users to define workflows as Python code (DAGs) that are parsed, validated, and compiled into an internal task graph representation. The system uses dynamic Python execution to instantiate DAG objects from .py files in the DAG folder, extracting task dependencies through operator instantiation and bitshift operators (>> and <<). DAG serialization converts the graph into JSON for storage in the metadata database, enabling stateless scheduler restarts and multi-scheduler deployments.

Executes tasks across distributed workers using a pluggable executor architecture that abstracts the underlying compute infrastructure. The system supports LocalExecutor (single machine), CeleryExecutor (distributed via message broker), KubernetesExecutor (pod-per-task), and custom executors. Tasks are queued with metadata, workers poll for assignments, and execution results are reported back via XCom (cross-communication) to the metadata database. The Supervisor process manages task lifecycle on each worker, spawning task runner subprocesses and capturing logs.

Provides built-in monitoring and alerting for DAG runs and task instances. SLA (Service Level Agreement) definitions on DAGs and tasks trigger alerts when execution exceeds time thresholds. The system integrates with external alerting systems (email, Slack, PagerDuty) via callback functions. Metrics are exposed in Prometheus format for integration with monitoring stacks. Deadline-based scheduling allows enforcing hard deadlines with automatic alerting. Task retry logic with exponential backoff provides automatic recovery from transient failures.

Enables running multiple versions of the same DAG simultaneously, allowing zero-downtime DAG updates. When a DAG definition changes, Airflow creates a new version while keeping the old version active for in-flight runs. The system tracks DAG version in the database, allowing queries to return results for specific versions. This enables gradual rollout of DAG changes: new runs use the new version while old runs continue with the old version. Version cleanup policies prevent unbounded growth of old versions.

Extensibility mechanism allowing developers to create custom operators, hooks, executors, and other Airflow components without modifying core code. Plugins are discovered via entry points or by placing Python files in the plugins directory. The system provides base classes (BaseOperator, BaseHook, BaseExecutor) that plugins extend. Custom plugins are automatically registered and available in DAG definitions. This enables organizations to build proprietary operators for internal systems.

Enables defining Service Level Agreements (SLAs) for tasks and DAGs, with automatic monitoring and alerting when SLAs are breached. SLAs are defined as timedelta values (e.g., task must complete within 1 hour of execution_date). The scheduler evaluates SLAs at each heartbeat and triggers alert callbacks when deadlines are missed. Supports custom alert handlers (email, Slack, webhooks) via callback functions.

Uses a relational database (PostgreSQL, MySQL, SQLite) to persist all Airflow state: DAG definitions, task instances, execution history, connections, and variables. The database schema includes tables for dag, dag_run, task_instance, xcom, log, and connection. State is serialized to JSON for complex objects (DAG definitions, task parameters). The scheduler can recover from crashes by querying the database for incomplete tasks and resuming execution.

The SchedulerJobRunner process continuously parses DAG files, evaluates scheduling rules (cron expressions, asset dependencies, deadlines), and instantiates DagRun objects when conditions are met. For each DagRun, the scheduler traverses the task dependency graph, evaluates task-level scheduling rules, and queues TaskInstance objects to the executor's queue. The scheduler uses a heartbeat-based loop (default 1s) with database-backed state to track which DagRuns and TaskInstances have been processed, enabling recovery after restarts. Asset-based scheduling allows DAGs to trigger when upstream datasets (assets) are updated.

Enables long-running tasks (e.g., waiting for external API responses, sensor polling) to defer execution and free up worker slots. When a task calls defer(), it saves its state and yields control back to the scheduler. The TriggererJobRunner process runs in a separate JVM/process, managing thousands of deferred tasks efficiently using async I/O (asyncio). When a trigger condition is met (e.g., external event received), the triggerer resumes the task on a worker. This pattern avoids blocking worker processes on I/O-bound operations.

Allows a single task definition to expand into multiple parallel task instances based on runtime data (e.g., list of files, query results). The expand() method takes a parameter name and an iterable (from XCom, task output, or literal list), creating one TaskInstance per item. The scheduler evaluates the iterable at runtime, generates task instances, and queues them for parallel execution. Downstream tasks can consume mapped task outputs via special XCom syntax, automatically aggregating results across all mapped instances.

Provides a lightweight publish-subscribe mechanism for tasks to share data via the metadata database. Tasks push values to XCom using task_instance.xcom_push(), and downstream tasks retrieve them via task_instance.xcom_pull(). XCom values are JSON-serialized and stored in the database, with automatic cleanup after DAG run completion. The system supports templating in task parameters (e.g., {{ task_instance.xcom_pull(task_ids='upstream_task') }}) to inject upstream results into task configuration.

Exposes Airflow functionality via a FastAPI-based REST API with OpenAPI (Swagger) specification. The API provides endpoints for DAG management (list, trigger, pause), DAG run inspection (status, logs), task instance queries, and XCom retrieval. The system uses OpenAPI-first development, generating API documentation and client SDKs from OpenAPI specs. Authentication is pluggable (basic auth, LDAP, OAuth) via Flask-AppBuilder (FAB) integration. The Execution API (separate from main REST API) provides low-latency task execution feedback for distributed task runners.

Provides a React-based web interface for monitoring and managing Airflow deployments. The UI displays DAG definitions, DAG run history with status visualization, task instance logs, and XCom values. Features include DAG triggering, task retry, and pause/unpause controls. The system supports internationalization (i18n) with translations for multiple languages. The UI communicates with the REST API, enabling real-time updates and responsive interactions. Role-based access control (RBAC) via Flask-AppBuilder restricts UI access based on user roles.

Airflow's extensibility model via provider packages that bundle operators, hooks, and sensors for specific platforms (AWS, GCP, Kubernetes, Spark, etc.). Providers are independently versioned Python packages that register with Airflow via entry points. Each provider includes operators (task implementations), hooks (reusable connection logic), and sensors (polling for conditions). The system uses a metadata registry to discover available providers and their capabilities. Custom providers can be developed by third parties and published to PyPI.

Provides Kubernetes-first deployment model via Helm charts and KubernetesExecutor. Each task executes in its own Kubernetes pod, enabling resource isolation, automatic scaling, and integration with Kubernetes RBAC and networking. The system includes Helm charts for deploying scheduler, workers, and supporting services (PostgreSQL, Redis) as Kubernetes resources. Pod templates are customizable, allowing per-task resource requests, node affinity, and image overrides. The KubernetesExecutor watches pod status and reports results back to the scheduler.