The Stack v2

Aggregates 67 TB of source code from the Software Heritage archive with automated license classification and filtering to retain only permissively licensed content (Apache 2.0, MIT, BSD, GPL variants, etc.). Uses metadata-driven filtering pipelines to exclude proprietary and restrictive licenses, enabling legal compliance for model training without manual license auditing. Implements a Software Heritage integration layer to access the largest open-source repository snapshot available.

Implements a rigorous deduplication pipeline that identifies and removes duplicate code across 600+ programming languages using content-based hashing and semantic similarity detection. Normalizes code formatting, whitespace, and comments to identify near-duplicates that would otherwise inflate dataset size and introduce training bias. Uses language-specific tokenization and AST-aware comparison for structural duplicates, not just string matching.

Applies automated PII detection pipelines to identify and redact sensitive information (email addresses, API keys, credentials, personal names, phone numbers, etc.) from source code before dataset release. Uses pattern matching, regex-based detection, and potentially ML-based classifiers to find PII in comments, strings, and code. Implements configurable redaction strategies (masking, removal, replacement with placeholders) while preserving code functionality.

Implements a governance framework allowing repository owners to request exclusion of their code from the dataset via an opt-out mechanism (e.g., registry, email contact, automated API). Processes exclusion requests, removes matching repositories from the dataset, and maintains an exclusion list for future dataset versions. Respects developer autonomy and copyright concerns while maintaining dataset openness by default.

Covers source code across 600+ programming languages with language-specific metadata (syntax, paradigm, ecosystem, file extensions, etc.). Implements language detection and classification pipelines to identify code language, extract language-specific features, and organize data by language family. Enables language-stratified sampling and analysis, supporting diverse model training use cases from general-purpose to language-specific code models.

Preserves and enriches repository-level metadata including creation date, last update, star count, fork count, contributor count, license type, and language distribution. Maintains file-to-repository mappings and directory structure information, enabling context-aware model training that understands code within its repository ecosystem. Implements metadata aggregation from Software Heritage and GitHub APIs to provide rich contextual information for each code sample.

Integrates with the Software Heritage archive, a comprehensive snapshot of open-source software repositories worldwide, to access code at scale without relying on individual repository APIs or GitHub. Implements Software Heritage API clients and data export pipelines to retrieve code content, metadata, and version history. Enables reproducible dataset snapshots by referencing specific Software Heritage revisions, supporting dataset versioning and reproducibility.

Implements versioning and release management for dataset versions (v1, v2, etc.) with documented changes, improvements, and data quality enhancements between versions. Maintains version-specific documentation, changelog, and reproducibility information. Enables users to select specific dataset versions for training, ensuring reproducibility and allowing comparison of model performance across dataset versions.

Integrates with Hugging Face Datasets library, providing standardized dataset loading, streaming, and sampling interfaces. Implements dataset cards with documentation, license information, and usage guidelines. Enables efficient streaming access to 67 TB dataset without downloading entire dataset, supporting memory-constrained training environments. Provides dataset splits, sampling strategies, and preprocessing utilities for common training workflows.

Serves as the primary training dataset for StarCoder2 models and other code generation models. Provides high-quality, permissively-licensed, deduplicated code across 600+ languages with repository context. Enables training of state-of-the-art code LLMs that understand diverse programming paradigms, languages, and coding patterns. Documented as essential resource for reproducing StarCoder2 and training similar models.