Repository Analytics And Statistics With Language And Contributor Analysis

via “token-level-dataset-statistics-and-composition-analysis”

6.3T token multilingual dataset across 167 languages.

Unique: Pre-computes and exposes language-level token statistics through Hugging Face Datasets metadata API, allowing users to query composition without downloading the full corpus — most datasets provide only total token counts or require users to scan the full dataset to understand language distribution

vs others: Faster and more convenient than analyzing raw mC4 or OSCAR directly, and more granular than summary statistics, enabling data-driven decisions about language weighting and sampling without custom preprocessing