Open Source Reproducible Data Processing Pipeline

via “open-source reproducible data processing pipeline”

30 trillion token web dataset with 40+ quality signals per document.

Unique: Publishes complete, open-source processing scripts enabling full reproducibility and transparency of data processing methodology. Users can inspect, verify, and reapply the pipeline to new data, unlike proprietary datasets where processing is opaque.

vs others: Open-source pipeline enables reproducibility and auditability vs. proprietary datasets (C4, Refinedweb) where processing methodology is proprietary or partially documented; enables research on data processing methodology itself.