Dolma

Aggregates 3 trillion tokens from 7 heterogeneous sources (Common Crawl, The Stack, peS2o, Project Gutenberg, Wikipedia, Wikibooks, C4) into a unified pretraining dataset with published filtering rules, deduplication strategies, and source mixing ratios. The assembly process applies source-specific quality filters and fuzzy deduplication via Duplodocus before combining sources at documented proportions, enabling reproducible dataset composition for LLM training.

Applies ultra-efficient fuzzy deduplication across the 3 trillion token corpus using the Duplodocus tool, which identifies and removes near-duplicate documents within and across source domains without requiring exact string matching. The fuzzy matching approach reduces redundancy while preserving legitimate diversity, operating at scale to handle the full dataset volume without prohibitive computational overhead.

Applies domain-specific quality filters and cleaning rules to each of the 7 source corpora using the Datamap-rs tool, which performs large-scale text normalization, content filtering, and quality assessment. The tool enables source-specific filtering strategies (e.g., code quality metrics for The Stack, academic rigor for peS2o) while maintaining computational efficiency across the full 3 trillion token dataset.

Provides multiple pretraining dataset variants (Standard Pool, Long Context Mix) with different source mixing ratios optimized for different training objectives. The variants are pre-composed and documented, allowing researchers to select a dataset variant matching their training goals without manually adjusting source proportions. The composition strategy reflects decisions about optimal balance between web text, code, academic content, and other domains.

Enables researchers to trace model outputs back to specific training documents and source domains using the OlmoTrace tool, which maps model predictions to the training data that influenced them. This capability supports interpretability research, bias analysis, and data attribution by linking model behavior to specific training examples and sources within the Dolma corpus.

Identifies and removes test set data from the pretraining corpus using the Decon tool, which detects overlap between training data and evaluation benchmarks. This prevents data leakage that would artificially inflate model performance on standard benchmarks, ensuring that reported model performance reflects genuine capability rather than memorization of test examples.

Integrates Dolma with the OlmoCore training framework, which provides fast, easy configuration for pretraining language models with documented data composition, hyperparameters, and training procedures. The framework enables researchers to reproduce model training exactly by specifying dataset variant, mixing ratios, and training configuration, supporting fully reproducible LLM development from data through model weights.

Provides the OLMES utility for running reproducible evaluations on models trained with Dolma and OlmoCore, enabling standardized benchmark testing with documented evaluation procedures. The utility ensures consistent evaluation methodology across research teams and model variants, supporting fair performance comparisons and preventing evaluation methodology drift.

Provides separate post-training corpora (distinct from the pretraining Dolma dataset) for instruction tuning and preference optimization, enabling researchers to fine-tune base models trained on Dolma with supervised instruction-following and reinforcement learning from human feedback (RLHF). The post-training data is composed and documented separately from pretraining data, supporting the full pipeline from base model training through instruction-tuned and preference-optimized variants.

Supports training the OLMo family of language models with multiple documented variants (7B, 32B base models; Instruct and Think configurations) using Dolma pretraining data and OlmoCore framework. Each model variant is trained with published hyperparameters, data composition, and training procedures, enabling researchers to reproduce or extend the model family with full transparency.