Minhash Based Deduplication At Petabyte Scale

via “document-level deduplication with hash-based matching”

30 trillion token web dataset with 40+ quality signals per document.

Unique: Uses document-level hash-based deduplication (preserving document boundaries) rather than token-level or fuzzy matching, enabling reproducible filtering and transparent deduplication hashes that users can inspect and verify. Processes 84 CommonCrawl dumps with consistent deduplication methodology.

vs others: Document-level deduplication is more interpretable and reproducible than token-level approaches, and the published deduplication hashes enable users to understand and verify which documents were removed, unlike proprietary datasets that hide deduplication decisions.

via “content-based deduplication at file and repository levels”

67 TB permissively licensed code dataset across 600+ languages.

Unique: Two-stage deduplication combining exact hash matching with fuzzy similarity matching (likely MinHash or Jaccard) to catch both identical and near-identical code — more thorough than single-stage approaches but computationally expensive

vs others: More aggressive deduplication than CodeSearchNet (which uses simple hash matching) because it catches near-duplicates, but less semantic than clone detection tools (which understand code structure) because it's content-based

via “minhash-based deduplication at petabyte scale”

Hugging Face's 15T token dataset, new standard for LLM training.

Unique: Uses MinHash locality-sensitive hashing for memory-efficient duplicate detection across 15 trillion tokens, avoiding the need to store full document hashes or maintain a global hash table. This enables processing at petabyte scale where naive approaches would exhaust available memory.

vs others: More memory-efficient than exact deduplication (which requires storing full hashes) and faster than string-similarity-based approaches (which require pairwise comparisons), making it practical for web-scale datasets where C4 and similar datasets use simpler or less effective deduplication strategies.

via “exact and fuzzy duplicate detection and removal”

Dataset by allenai. 7,61,810 downloads.

Unique: C4 combines exact and fuzzy deduplication in a two-stage pipeline, using MinHash for efficient approximate matching at scale. The approach is fully reproducible and the thresholds are published, allowing researchers to audit or adjust deduplication aggressiveness. This is more sophisticated than simple exact-match deduplication but simpler than learned semantic deduplication models.

vs others: C4's two-stage deduplication is more scalable and transparent than semantic deduplication models, while catching more duplicates than exact-match-only approaches, making it practical for petabyte-scale datasets.