Metaflow

Define ML pipelines as directed acyclic graphs by subclassing FlowSpec and decorating Python methods with @step. Metaflow parses the class structure to build a dependency graph, automatically determining task execution order and parallelization opportunities. The framework handles step-to-step data passing through a content-addressed artifact store, enabling reproducible, versioned workflows without explicit orchestration code.

Automatically snapshot all step outputs (artifacts) into a content-addressed store (TaskDataStore, FlowDataStore) keyed by content hash. Each run is immutable and fully reproducible — artifacts are versioned by their hash, not by timestamp or run ID. Supports local filesystem storage for development and S3/cloud backends for production, with transparent serialization of Python objects (pickle, JSON, Parquet).

Execute flows programmatically using Runner and NBRunner classes, enabling integration with notebooks, scripts, or external orchestrators. Runner executes flows locally or on configured backends, returning ExecutingRun objects for monitoring. Supports programmatic parameter passing, environment variable injection, and result retrieval. NBRunner is optimized for Jupyter notebooks with inline execution and progress tracking.

Provide S3-native utilities for reading, writing, and managing data in S3 without downloading to local disk. S3 tools support streaming reads/writes, multipart uploads, and efficient data transfer. Integrates with artifact storage, allowing flows to work with large datasets (>100GB) without memory overhead. Supports S3 Select for querying Parquet/CSV files server-side, reducing data transfer.

Metaflow provides S3 tools (S3 class, S3Client) for reading and writing data to S3 within flow steps. The S3 integration handles authentication via IAM roles, supports both local and cloud execution, and provides efficient data transfer with progress tracking. Data can be stored in S3 as artifacts or accessed directly from steps, enabling scalable data pipelines without local storage constraints.

Execute flows on local machine, AWS Batch, Kubernetes, or cloud-native services (AWS Step Functions) through a pluggable runtime abstraction. The @batch, @kubernetes, and @step_functions decorators specify compute requirements per step (CPU, memory, GPU, timeout). Metaflow translates these to cloud-native job definitions, handling image building, credential injection, and result retrieval automatically.

Specify isolated Python environments per step using @conda, @pypi, or @uv decorators with dependency specifications. Metaflow builds or resolves environments at runtime, installing packages into isolated containers or virtual environments. Supports environment caching to avoid redundant builds, and 'environment escape' for system-level dependencies (CUDA, system libraries). Each step runs in its declared environment, enabling dependency isolation and version pinning.

Query and inspect completed runs using Flow, Run, Step, Task, and DataArtifact client classes. Access any run's metadata (status, timestamps, parameters), step outputs, and task logs without re-executing. The API supports filtering, iteration, and programmatic access to artifacts, enabling post-hoc analysis, debugging, and integration with notebooks or dashboards. Metadata is stored in a pluggable provider (LocalMetadataProvider, ServiceMetadataProvider) for local or remote access.

Convert Metaflow flows to production-grade orchestrator definitions using Deployer API and @argo_workflows or @step_functions decorators. Metaflow generates Argo Workflow YAML or AWS Step Functions state machines from the flow DAG, handling step-to-step data passing, error handling, and retry logic. Supports event-driven triggers (Argo Events) and scheduled execution via cron or external events.

Define flow parameters using @parameter decorator with type hints, default values, and help text. Metaflow validates parameter types at runtime and exposes them via CLI arguments or programmatic APIs. Supports complex types (lists, dicts, JSON), file inclusion via @include_file, and deploy-time field injection for secrets or environment-specific values. Parameters are versioned with each run, enabling reproducibility and parameter sweeps.

Generate interactive HTML cards for visualizing step outputs using @card decorator and Card API. Cards render plots, tables, markdown, and custom components directly in Metaflow UI or exported as standalone HTML. Supports multiple card types (plot, table, markdown, custom) with lazy rendering to avoid memory overhead. Cards are stored alongside artifacts, enabling rich result exploration without external dashboards.

Extend Metaflow via plugin architecture supporting custom compute backends, metadata providers, datastores, and decorators. Plugins are discovered via entry points and loaded dynamically, allowing third-party integrations without modifying core code. The extension_support module provides base classes (FlowDecorator, MetadataProvider, DataStore) for implementing custom functionality. Plugins can override default behavior (e.g., custom S3 client, alternative metadata storage).

Track flow execution metadata (run ID, status, parameters, timestamps, task logs) using pluggable metadata providers. LocalMetadataProvider stores metadata in local filesystem; ServiceMetadataProvider connects to remote metadata service. Metadata includes run lineage, step dependencies, task status, and execution times. Enables querying run history, comparing runs, and debugging via CLI commands (metaflow show, metaflow logs).