dvc

DVC tracks large files and datasets by storing metadata (.dvc files) in Git while maintaining actual data in a content-addressed object database (cache layer). Uses SHA256 hashing to deduplicate data across versions and projects, enabling efficient storage without bloating Git repositories. The Repo class coordinates between Git's SCM layer and DVC's FileSystem abstraction to transparently manage data lifecycle.

DVC pipelines are defined in dvc.yaml using a declarative YAML format where each stage specifies dependencies (inputs), commands (execution), and outputs (results). The Index and Graph System builds a directed acyclic graph (DAG) from stage definitions, enabling DVC to compute execution order, detect changes, and run only affected stages. The Stage class encapsulates command execution with dependency tracking, while the Output system manages stage artifacts.

DVC's Cache and Object Database system stores data using content-addressed storage (SHA256 hashes as keys), enabling automatic deduplication across versions and projects. The CacheManager handles cache operations (add, retrieve, verify), while the object database maintains the actual cached files organized by hash. Garbage collection removes unreferenced cache entries, and cache integrity is verified through hash validation.

DVC's Index and Graph System builds a directed acyclic graph (DAG) from stage definitions, tracking dependencies between stages and detecting which stages need re-execution when inputs change. The Index class maintains the graph structure and provides methods for traversal and change detection. This enables efficient incremental execution by identifying affected stages without re-running the entire pipeline.

DVC provides a comprehensive CLI through the dvc.cli module with subcommands for all major operations (add, run, push, pull, repro, etc.). The CLI uses argparse for argument parsing and provides consistent help/error messages across commands. Each subcommand is implemented as a separate module with a run() method, enabling modular command implementation and testing.

DVC exposes a Python API through the dvc.api module and Repo class, enabling programmatic access to all DVC operations without CLI invocation. The API provides methods for data operations (add, push, pull), pipeline management (run, repro), and experiment tracking. This enables integration with Jupyter notebooks, custom scripts, and external tools.

DVC abstracts storage operations through a FileSystem abstraction layer that supports S3, GCS, Azure Blob Storage, HDFS, and local paths. The Remote Storage Operations subsystem handles push/pull operations with configurable remote endpoints defined in .dvc/config. Data is transferred using the CacheManager, which manages local cache coherency and remote synchronization, enabling teams to share data without direct file system access.

DVC's Experiment Management subsystem enables running multiple ML experiments with different parameters/code versions, tracked in a queue system with configurable executors. The Experiment Lifecycle manages experiment creation, execution, and storage, while the Collection system organizes results for comparison. Experiments are stored as Git branches or commits, enabling version control of entire experiment runs including code, parameters, and outputs.

DVC tracks model metrics (accuracy, loss, etc.) and pipeline parameters (learning rate, batch size, etc.) from files (JSON, YAML, CSV) specified in dvc.yaml. The Metrics and Parameters subsystem parses these files and enables comparison across experiments and pipeline runs. The Plots System generates visualizations from metrics data, supporting multiple plot types (line, scatter, confusion matrix) with automatic rendering in compatible tools.

DVC enables importing data from external repositories using the External Repository Integration subsystem, which clones remote DVC repos and extracts specific data files/versions. The import operation creates a dependency on the external repo, automatically pulling updates when the external repo changes. This is implemented through the dependency/repo.py module, which handles external repo resolution and data fetching.

DVC's State Tracking subsystem maintains a local state database (.dvc/tmp/dvc-state) that records file modification times, sizes, and hashes to detect when data has changed without re-hashing. The CacheManager uses this state information to determine if cached files are still valid or need re-computation. This enables efficient incremental pipeline execution by skipping stages whose inputs haven't changed.

DVC's Configuration System manages settings through multiple layers: system-wide (/etc/dvc/config), user-level (~/.config/dvc/config), and repository-level (.dvc/config) with clear precedence rules. The Config class parses YAML/INI configuration files and provides unified access to settings like remote storage endpoints, cache location, and execution parameters. Configuration can be modified via CLI commands (dvc config) or direct file editing.

DVC's Repository Management subsystem handles repo initialization (dvc init), which creates the .dvc directory structure with config, cache, and metadata files. The Repo class serves as the central coordinator for all operations, managing initialization state, configuration loading, and lifecycle events. Repository initialization integrates with Git, creating .dvc/.gitignore to exclude cache from version control.

DVC's Filesystem Abstraction layer provides a unified interface for accessing data across different storage backends (local, S3, GCS, Azure, HDFS) through a common API. The abstraction handles protocol-specific details (authentication, path normalization, error handling) transparently, allowing higher-level components to work with any storage backend without modification. This is implemented through pluggable filesystem classes that inherit from a common base.