DVC

DVC versions large files and ML models by computing content hashes (checksums) and storing metadata (.dvc files) in Git while keeping actual data in local cache or remote storage. Uses a Repo class that coordinates cache management, remote synchronization, and Git integration to enable data versioning without bloating the Git repository. The Output class associates files with their checksums and manages retrieval from content-addressable storage, enabling efficient deduplication across experiments and team members.

DVC pipelines are defined as directed acyclic graphs (DAGs) where each Stage represents a computational step with explicit dependencies (inputs) and outputs. The Stage class tracks command execution, input/output relationships, and reproduction status. The Repo class maintains a pipeline index that resolves dependency chains, enabling DVC to determine which stages need rerunning when inputs change. Pipeline definitions are stored in dvc.yaml files, making them version-controllable and shareable.

DVC integrates deeply with Git through an SCM (Source Control Management) abstraction that enables tracking .dvc metadata files, reading Git history, and managing experiment branches. The SCM class provides methods to commit files, create branches, read commit history, and resolve Git conflicts. This integration allows DVC to store pipeline definitions and metadata in Git while keeping large data files separate. The experiment system leverages Git branching to create isolated experiment variants without polluting the main branch.

DVC stores configuration in .dvc/config files using INI format, supporting hierarchical configuration (system, global, local, project-level). The Configuration class parses these files and merges settings from multiple levels, with local settings overriding global settings. Configuration includes remote storage URLs, cache settings, authentication credentials, and pipeline parameters. This design enables teams to share project-level config (remotes, cache settings) via Git while keeping sensitive credentials in local .dvc/config.local files (which are .gitignored).

DVC exposes a Python API through the Repo class that enables developers to programmatically perform DVC operations (add data, run pipelines, track experiments) without using the CLI. The API provides methods like repo.add(), repo.run(), repo.reproduce(), and repo.experiments.run() that mirror CLI commands. This enables integration with Jupyter notebooks, custom scripts, and external tools. The API is built on the same core components as the CLI (Repo, Stage, Output classes), ensuring consistency between programmatic and CLI usage.

DVC provides status and diff commands that compare current workspace state against cached/committed state. The status command shows which files have changed, which stages need rerunning, and which experiments have uncommitted results. The diff command compares parameters and metrics across Git commits or experiments, showing which values changed and by how much. These commands use the checksum-based tracking system to detect changes efficiently without recomputing hashes.

DVC implements intelligent pipeline reproduction by computing checksums of stage inputs (code, data, parameters) and comparing against cached results. The Repo class maintains a cache index that tracks which outputs correspond to which input states. When a stage's dependencies change, DVC detects this via checksum mismatch and marks only affected downstream stages for rerunning. This avoids redundant computation while guaranteeing reproducibility because outputs are tied to specific input states.

DVC abstracts storage backends (S3, GCS, Azure Blob, HDFS, SSH, local paths) through a unified Remote Storage interface. The Repo class manages remote configuration and coordinates push/pull operations that synchronize data between local cache and remote storage. Remote storage is configured in .dvc/config files and supports authentication via environment variables or credential files. This enables teams to store large files in cloud buckets while keeping local workspaces clean, with automatic deduplication across users.

DVC's experiment system tracks ML experiments by capturing parameters (hyperparameters, configuration) and metrics (accuracy, loss, F1) from runs. Parameters are read from YAML/JSON files specified in dvc.yaml, while metrics are extracted from output files (JSON, CSV, YAML). The Experiment class queues and executes experiment variants, storing results in a local Git-based experiment registry. Experiments are compared via a diff system that shows parameter and metric changes across runs, enabling data-driven model selection.

DVC enables parameterized pipelines where stage commands reference variables from params.yaml or other parameter files. Parameters are injected into stage commands at execution time, allowing the same pipeline definition to run with different configurations. The Experiment system extends this with parameter sweeping — automatically generating experiment variants by iterating over parameter ranges or grids. This is implemented through the Experiment queue, which creates multiple experiment branches with different parameter values.

DVC generates visual representations of pipeline DAGs showing stage dependencies, inputs, and outputs. The visualization system parses dvc.yaml and builds a dependency graph, then renders it as a directed graph (typically in Mermaid or Graphviz format). This enables developers to understand data lineage, identify bottlenecks, and verify pipeline structure. The diff system also visualizes how pipeline structure changes across Git commits.

DVC extracts metrics (scalar values like accuracy, loss) and plots (data for visualization like confusion matrices, ROC curves) from training outputs in multiple formats (JSON, CSV, YAML, TSV). The Metrics class parses these files and stores them in the experiment registry. Plots are rendered as interactive visualizations (line charts, scatter plots, confusion matrices) in the DVC UI or exported as static images. This enables teams to compare model performance across experiments without manually parsing output files.

DVC experiments are stored as Git commits with metadata (parameters, metrics) attached, enabling full reproducibility and version history. When an experiment is queued, DVC creates a new Git branch with modified parameter files and stage outputs. The experiment registry tracks which Git commits correspond to which experiments, enabling developers to checkout a specific experiment's code and data state. This design ensures experiments are reproducible because all inputs (code, data, parameters) are captured in Git history.

DVC abstracts file system operations through a FileSystem interface that supports local paths, cloud storage (S3, GCS, Azure), and remote protocols (SSH, HDFS). This abstraction enables DVC to treat all storage backends uniformly — operations like read, write, exists, and list work identically whether the path is local or remote. The abstraction is implemented through provider-specific classes (LocalFileSystem, S3FileSystem, etc.) that inherit from a common base. This design enables DVC to support new storage backends by implementing the FileSystem interface without modifying core logic.