Curated Dataset Provision With Domain Context And Preprocessing Guidance

via “domain-specific dataset curation and subset extraction”

1.2M image-text pairs with GPT-4V captions.

Unique: Enables systematic curation of domain-specific subsets from 1.2M images using GPT-4V captions as semantic filters, allowing extraction of specialized datasets without manual domain annotation or external labeling services

vs others: More flexible than fixed domain-specific datasets (e.g., medical imaging datasets) which are typically small and expensive to create; leverages rich caption semantics for more accurate domain filtering than keyword-based approaches

via “multi-domain pretraining corpus assembly”

EleutherAI's 825 GiB diverse training dataset from 22 sources.

Unique: Pioneered the multi-domain curation approach by intentionally combining 22 diverse, high-quality subsets (academic papers, books, code, web, specialized sources) rather than scraping a single massive web corpus. This architectural choice prioritizes knowledge breadth and domain coverage over raw scale, influencing the design of subsequent open datasets like LAION, RedPajama, and Falcon-Refinedweb.

vs others: Broader domain coverage than Common Crawl-only datasets (e.g., C4) and higher quality than raw web scrapes due to curation of academic, code, and book sources; smaller than Falcon-Refinedweb (1.5T tokens) but more carefully curated and widely adopted as a benchmark for model evaluation

via “domain-aware-document-filtering-and-balancing”

6.3T token multilingual dataset across 167 languages.

Unique: Applies domain-aware filtering that balances representation across content types (news, academic, social media, forums) rather than treating all domains equally or using only global quality thresholds

vs others: More balanced than raw web crawls (which are dominated by news and social media); more principled than naive domain filtering by using explicit domain classification and configurable balancing targets

via “multi-source pretraining data composition with documented curation rules”

Allen AI's 3T token dataset for fully reproducible LLM training.

Unique: Dolma's distinguishing feature is comprehensive documentation of data curation decisions (exact filtering rules, deduplication methods via Duplodocus, mixing ratios) released alongside trained models (OLMo 7B, 32B), enabling full reproducibility. Most pretraining datasets (C4, The Pile, ROOTS) document composition at a high level but not the specific algorithmic rules applied. Dolma's integration with OlmoTrace enables tracing model outputs back to source training documents, providing data provenance that most datasets lack.

vs others: Dolma provides greater transparency and reproducibility than C4 or The Pile through documented filtering rules and deduplication specifications, while offering more diverse source coverage (code + academic + literary) than web-only datasets like C4, though it is smaller than ROOTS (1.6T vs 3T tokens) and less frequently updated than continuously-refreshed web crawl datasets.

via “news-domain-specific text variant with distribution matching”

Google's cleaned Common Crawl corpus used to train T5.

Unique: Applies domain-specific filtering heuristics to C4 to create a news-distribution-matched subset, enabling news-focused pre-training without separate news corpus collection; maintains consistency with C4 cleaning pipeline while adding domain-specific selection

vs others: Simpler and more reproducible than collecting news from multiple sources; smaller and more focused than full C4, but may lack editorial quality and fact-checking standards of professional news datasets

via “dataset loading and preprocessing for heterogeneous task formats”

Implementation of a paper on Multiagent Debate

Unique: Implements task-specific dataset loaders that normalize heterogeneous formats (GSM JSON, MMLU CSV, biography articles, generated math) into consistent input structures, abstracting format differences from debate generation logic

vs others: More specialized than generic data loading libraries because it understands task-specific semantics (e.g., extracting questions and ground truth from domain-specific formats) rather than treating all datasets as generic CSV/JSON

via “document-domain dataset sampling and filtering”

Dataset by mlfoundations. 8,57,357 downloads.

Unique: Provides streaming access with metadata-based filtering on trillion-token dataset without requiring full download, using Hugging Face Datasets infrastructure for efficient subset construction. Enables on-demand domain-specific corpus creation from larger collection.

vs others: More flexible than fixed-size domain datasets (e.g., ArXiv papers, legal documents) by allowing dynamic filtering from larger corpus; more efficient than downloading full dataset for subset access.

via “dataset curation and quality assessment for fine-tuning”


Unique: Emphasizes the critical but often-overlooked role of data quality in fine-tuning success, with practical techniques for identifying distribution shifts and measuring dataset characteristics that predict model performance

vs others: More rigorous than ad-hoc data preparation while remaining practical for teams without dedicated data engineering resources; focuses on fine-tuning-specific quality metrics rather than generic data cleaning

robust introduction to the subject and also the foundation for a Data Analyst “nanodegree” certification sponsored by Facebook and MongoDB.

via “data-curation-and-filtering”