CulturaX

Performs exact and fuzzy deduplication across 167 languages on 6.3 trillion tokens by combining mC4 and OSCAR source datasets using language-agnostic hashing and probabilistic data structures. Implements document-level and paragraph-level deduplication with configurable thresholds to remove redundant training data while preserving linguistic diversity across low-resource languages.

Applies multi-stage quality filtering combining content-based heuristics (text length, language detection confidence, character distribution) and metadata-based signals (domain reputation, crawl freshness) to remove low-quality documents across 167 languages. Uses language-aware tokenization to compute quality metrics that account for morphological and script differences between language families.

Merges mC4 and OSCAR datasets while resolving conflicts (duplicate documents from both sources, conflicting metadata, version mismatches) using a priority-based merge strategy that preserves the highest-quality version of each document. Implements source-aware deduplication that tracks which source contributed each document and resolves overlaps by selecting the version with better quality signals.

Enables extraction of language-specific subsets from the full 167-language corpus with configurable sampling strategies (uniform, stratified by quality, weighted by language family) to support language-specific model training or analysis. Provides statistics on token distribution, document counts, and quality metrics per language to inform sampling decisions.

Maintains explicit versioning of the CulturaX dataset with documented deduplication and filtering parameters, enabling reproducible dataset reconstruction and tracking of which documents came from which source and processing step. Includes metadata for each document recording its source (mC4 vs OSCAR), deduplication status, quality scores, and processing pipeline version.

Implements language-aware sampling that prioritizes preservation and oversampling of low-resource languages (e.g., Icelandic, Maltese, Amharic) to prevent underrepresentation in multilingual model training. Uses language family groupings and token count analysis to identify underrepresented languages and applies weighted sampling to ensure minimum coverage thresholds.

Enables streaming access to the 6.3 trillion token dataset without downloading the full corpus, using Hugging Face Datasets streaming mode to load documents on-the-fly during training. Supports batching, shuffling, and caching strategies optimized for distributed training pipelines to minimize memory footprint while maintaining training efficiency.

Automatically detects language for each document and normalizes text across diverse writing systems (Latin, Cyrillic, Arabic, CJK, Indic scripts, etc.) to ensure consistent preprocessing across all 167 languages. Uses language detection models (fastText or similar) with confidence thresholding and script-aware normalization (Unicode normalization, diacritic handling) to handle multilingual text robustly.

Computes multi-dimensional quality scores for each document based on content properties (text length, language detection confidence, character distribution, readability metrics) and metadata signals (domain reputation, crawl freshness, source reliability). Enables ranking and filtering documents by quality without binary accept/reject decisions, supporting nuanced quality-based sampling.

Analyzes document source domains (news sites, academic papers, social media, forums, etc.) and applies domain-specific filtering rules to balance representation across content types. Prevents domain-specific biases (e.g., over-representation of news or Wikipedia) that could skew model behavior toward particular writing styles or information sources.