Scaling Laws

In depth

Empirical relationships showing that a language model's loss falls as a predictable power law as you increase parameters, training data, and training compute. Kaplan et al. at OpenAI measured these trends across more than seven orders of magnitude and found that architectural details like width and depth matter far less than scale. Labs use scaling laws to forecast how a larger training run will perform and to decide how to split a fixed compute budget between model size and data.

Primary source: arxiv.org