Training Compute-Optimal Large Language Models (Chinchilla)
Product* ⭐ 04/2022: [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan)](https://arxiv.org/abs/2204.01691)
Capabilities5 decomposed
compute-optimal model scaling with token-to-parameter ratio optimization
Medium confidenceDetermines the mathematically optimal allocation of training compute budget between model parameters and training tokens using empirical scaling laws derived from training runs across multiple model sizes. The approach fits power-law relationships to observed loss curves, then solves for the compute-optimal ratio where both parameters and tokens scale equally with total compute budget (N ≈ C/6L, D ≈ 20C/L where C is compute budget). This differs from prior Kaplan scaling laws which suggested undertrained models; Chinchilla shows equal parameter-token scaling is optimal.
Empirically derives compute-optimal scaling laws through systematic training of models from 70M to 540B parameters, discovering that parameter count and token count should scale equally with compute budget (contrary to prior Kaplan et al. scaling laws which suggested undertrained models were optimal). Uses power-law fitting to loss curves across multiple scales to establish generalizable relationships.
More compute-efficient than prior Kaplan scaling laws by ~20% through equal parameter-token scaling; provides empirically-grounded recommendations rather than theoretical extrapolations, making it more reliable for practical training budget allocation decisions
loss prediction across model scales via empirical scaling law interpolation
Medium confidencePredicts training loss for unseen model sizes by fitting power-law functions (L(N,D) = aN^α + bD^β + E) to loss measurements from trained models at multiple scales, then interpolating or extrapolating to new parameter/token combinations. The model captures how loss decreases with both parameter count and data size, enabling loss prediction without retraining. Chinchilla's key finding is that optimal loss follows L_opt(C) = E + (C/6L)^-α where both exponents are approximately -0.07.
Fits bidirectional power-law scaling laws (loss as function of both parameters AND tokens) rather than unidirectional extrapolation; discovers that optimal loss follows a specific compute-dependent curve where both parameter and token exponents are nearly identical (~-0.07), enabling unified compute-optimal recommendations.
More accurate than prior Kaplan scaling laws for predicting loss at new scales because it accounts for both parameter and token scaling simultaneously; enables loss prediction without retraining, saving weeks of compute compared to empirical validation
compute budget allocation solver for parameter-token tradeoff
Medium confidenceGiven a fixed training compute budget (measured in FLOPs), solves for the optimal split between model parameters (N) and training tokens (D) by applying the derived scaling law relationships. The solver uses the constraint that compute C ≈ 6ND (accounting for forward and backward passes) and the empirical finding that optimal allocation has N ≈ C/6L and D ≈ 20C/L, where L is the sequence length. This produces a deterministic recommendation for model size and dataset size given compute budget.
Solves the parameter-token allocation problem as a constrained optimization using empirically-derived scaling laws, producing deterministic recommendations rather than heuristics. The key insight is that equal scaling of parameters and tokens (N ∝ D ∝ √C) is optimal, contrary to prior assumptions of undertrained models.
Provides data-driven allocation recommendations vs rule-of-thumb approaches; accounts for both parameter and token scaling simultaneously rather than treating them independently, resulting in ~20% better compute efficiency than prior Kaplan-based approaches
empirical scaling law fitting and validation across model scales
Medium confidenceTrains multiple model instances at different scales (70M, 400M, 1B, 3B, 7B, 13B, 70B parameters) with varying token counts, measures training loss curves, and fits power-law functions to the observed data. The fitting process uses least-squares regression on log-log plots to extract scaling exponents and coefficients, then validates the fit by comparing predicted vs observed loss on held-out model sizes. This creates an empirical foundation for all downstream scaling law predictions and recommendations.
Conducts systematic empirical training across 6+ model scales from 70M to 540B parameters with multiple token counts per scale, fitting bidirectional power-law relationships rather than relying on theoretical extrapolation. Validates fits on held-out scales to ensure generalization.
More comprehensive than prior Kaplan et al. scaling law study by covering larger model sizes (up to 540B vs 1.3B) and testing both parameter and token scaling simultaneously; provides empirically-grounded exponents rather than theoretical predictions
training efficiency benchmarking and comparison across scales
Medium confidenceMeasures and compares training efficiency metrics (loss per compute unit, convergence speed, sample efficiency) across different model sizes and token counts. Efficiency is quantified as the loss achieved per unit of compute (FLOPs), enabling direct comparison of whether larger models or more tokens provide better returns on compute investment. The benchmarking reveals that compute-optimal allocation (equal parameter-token scaling) achieves better efficiency than either parameter-heavy or token-heavy alternatives.
Systematically benchmarks training efficiency across a wide range of model sizes (70M to 540B) and token counts, revealing that compute-optimal allocation (N ≈ D) achieves ~20% better efficiency than undertrained or overtrained alternatives. Provides empirical efficiency curves rather than theoretical predictions.
More comprehensive efficiency analysis than prior work by testing both parameter and token scaling; reveals that equal scaling is optimal, contradicting prior assumptions of undertrained models being more efficient
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with Training Compute-Optimal Large Language Models (Chinchilla), ranked by overlap. Discovered automatically through the match graph.
ultrascale-playbook
ultrascale-playbook — AI demo on HuggingFace
OPT
Open Pretrained Transformers (OPT) by Facebook is a suite of decoder-only pre-trained transformers....
OPT
Open Pretrained Transformers (OPT) by Facebook is a suite of decoder-only pre-trained transformers. [Announcement](https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/).
CS324 - Advances in Foundation Models - Stanford University
CS25: Transformers United V2 - Stanford University
GPTSwarm
Language Agents as Optimizable Graphs
Best For
- ✓ML researchers designing new LLM training runs
- ✓ML engineers planning compute budgets for foundation models
- ✓Teams deciding between scaling parameters vs data for fixed compute
- ✓ML researchers planning experiments and allocating compute budgets
- ✓Teams evaluating whether to scale up existing models
- ✓Organizations comparing multiple model architecture proposals
- ✓ML engineers planning training infrastructure and hardware procurement
- ✓Teams with fixed compute budgets (e.g., limited GPU hours or cloud credits)
Known Limitations
- ⚠Scaling laws derived from specific training setups (dense transformer architectures, particular optimization hyperparameters); may not transfer perfectly to sparse models, mixture-of-experts, or different training regimes
- ⚠Empirical fitting introduces uncertainty bounds; predictions degrade significantly beyond the range of observed model sizes (roughly 70M to 540B parameters in the paper)
- ⚠Does not account for inference cost optimization or downstream task performance variance; optimizes only for training loss
- ⚠Assumes homogeneous data quality and standard cross-entropy loss; does not model domain-specific or curriculum learning effects
- ⚠Accuracy degrades significantly outside the training range of observed scales; extrapolation beyond 540B parameters is unreliable
- ⚠Assumes smooth power-law relationships; does not capture phase transitions, emergent capabilities, or task-specific performance cliffs
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
* ⭐ 04/2022: [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan)](https://arxiv.org/abs/2204.01691)
Categories
Alternatives to Training Compute-Optimal Large Language Models (Chinchilla)
Are you the builder of Training Compute-Optimal Large Language Models (Chinchilla)?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →