Apache Airflow

Enables users to define workflows as Python code (DAGs) that are parsed, validated, and compiled into an internal task graph representation. The system uses Python's AST parsing and dynamic module loading to extract DAG objects from Python files in the dags_folder, serializing them into the metadata database with support for versioning and incremental updates. DAG serialization stores both the code structure and runtime metadata (schedule intervals, retries, dependencies) in JSON format to enable stateless scheduler execution.

The SchedulerJobRunner process continuously polls the metadata database to identify ready-to-execute tasks based on dependency resolution, scheduling constraints (cron/timetable expressions), and asset-based triggers. It implements a state machine for task instances (queued → scheduled → running → success/failed) and uses a priority queue to order task execution. The scheduler evaluates task dependencies (upstream/downstream relationships), XCom-based data dependencies, and asset-based deadlines to determine execution eligibility without requiring external orchestration services.

Provides production-ready Helm charts for deploying Airflow on Kubernetes, including scheduler, webserver, worker, and triggerer components as separate pods. Supports horizontal autoscaling of workers based on task queue depth (via KEDA or custom metrics). The KubernetesExecutor launches one pod per task, enabling fine-grained resource isolation and dynamic scaling. Includes sidecar containers for log collection and monitoring integration.

Enables developers to create custom operators, hooks, sensors, and executors by extending base classes and registering them as entry points. Providers are Python packages that bundle related integrations and are discovered via setuptools entry points. The plugin system supports custom macros, timetables, and authentication backends. Providers can define their own CLI commands and UI extensions.

Enables reprocessing historical data by creating DagRun instances for past dates and executing tasks with historical execution dates. The backfill command generates task instances for a date range and submits them to the executor. Supports parallel backfill execution (multiple workers processing different date ranges) and incremental backfill (skipping already-completed runs). Backfill respects task dependencies and SLAs, enabling safe historical reprocessing.

Enables defining Service Level Agreements (SLAs) for tasks and DAGs, with automatic monitoring and alerting when SLAs are breached. SLAs are defined as timedelta values (e.g., task must complete within 1 hour of execution_date). The scheduler evaluates SLAs at each heartbeat and triggers alert callbacks when deadlines are missed. Supports custom alert handlers (email, Slack, webhooks) via callback functions.

Uses a relational database (PostgreSQL, MySQL, SQLite) to persist all Airflow state: DAG definitions, task instances, execution history, connections, and variables. The database schema includes tables for dag, dag_run, task_instance, xcom, log, and connection. State is serialized to JSON for complex objects (DAG definitions, task parameters). The scheduler can recover from crashes by querying the database for incomplete tasks and resuming execution.

Airflow abstracts task execution through an Executor interface that supports multiple backends: LocalExecutor (single-machine), CeleryExecutor (distributed message queue), KubernetesExecutor (per-task pods), and SequentialExecutor (single-threaded). The scheduler submits tasks to the executor, which handles resource allocation, process/container lifecycle management, and result collection. The Execution API (FastAPI-based) provides a standardized protocol for task runners to report status, retrieve task definitions, and stream logs back to the scheduler.

Enables generating multiple task instances from a single task definition based on runtime data (e.g., list of files, database query results). Uses the expand() method to map over XCom values or task parameters, creating a task group with N instances. The scheduler evaluates the mapped task at runtime, creating task instances dynamically without requiring DAG code changes. Supports nested mapping and conditional task generation through custom mapping functions.

Allows long-running tasks to yield control back to the scheduler via the Triggerer process, which manages async I/O operations (polling APIs, waiting for webhooks) without blocking worker processes. Tasks use the defer() method to suspend execution and register a trigger (e.g., TimeDeltaTrigger, DateTimeTrigger, custom async triggers). The Triggerer polls triggers asynchronously and resumes tasks when conditions are met, reducing resource consumption for I/O-bound workflows.

Enables scheduling workflows based on data availability (assets) rather than time-based schedules. Assets are logical data entities (tables, files, datasets) that can be produced by tasks and consumed by downstream workflows. The scheduler tracks asset updates and triggers dependent workflows when upstream assets are updated. Implements automatic lineage tracking by analyzing task inputs/outputs, creating a data dependency graph visible in the UI.

Provides a key-value store (XCom table) for tasks to share small amounts of data (strings, numbers, JSON objects). Tasks push XCom values using xcom_push() and pull upstream values using xcom_pull(). XCom supports templating in task parameters, allowing downstream tasks to reference upstream outputs without explicit pull operations. Serialization is pluggable (JSON, pickle, custom serializers) and supports compression for larger payloads.

Provides a standardized Operator base class that encapsulates task logic and integrates with external systems (databases, cloud services, APIs). Operators are organized into provider packages (airflow-providers-*) that bundle related operators, hooks (connection wrappers), and sensors. The operator ecosystem includes BashOperator, PythonOperator, SQLOperator, KubernetesPodOperator, and specialized operators for cloud platforms (AWS, GCP, Azure). Hooks abstract connection management and API calls, enabling code reuse across operators.

Exposes a comprehensive REST API (OpenAPI-documented) for programmatic access to Airflow resources (DAGs, runs, task instances, logs). The Execution API (FastAPI-based) provides a standardized protocol for task runners to fetch task definitions, report execution status, and stream logs. The API supports RBAC (role-based access control) through Flask-AppBuilder, enabling multi-tenant deployments with fine-grained permissions.

Provides a React-based web interface for visualizing DAGs as directed acyclic graphs, monitoring task execution in real-time, and managing Airflow resources. The UI displays task dependencies, execution history, logs, and XCom values. Supports internationalization (i18n) for multiple languages. The UI is built with Flask-AppBuilder for authentication and authorization, and uses a REST API backend for data fetching.