2020
GPT-3, 175B
In-context learning: a couple of examples is enough.
Milestones on the line · SCALE
Scaling laws
It turned out you can predict a model's quality in advance from how much compute and data go into it. Progress stopped being luck and became a budgeting question.
Quality grows predictably with compute, data and parameters. That turned research into industrial planning and started the data-centre race.
StatusPASSED
TypeMilestones on the line
Marker2020
What exists today6
Researched
2020
Kaplan scaling laws
The curves clusters have been planned against ever since.
2022
Chinchilla
Data beats size — every training budget gets rebuilt.
2022
Emergent abilities
Some skills appear in a jump past a scale threshold rather than growing smoothly.
In progress
сейчас
Running out of data
High-quality human text on the internet is close to exhausted. Next come synthetic data, video and companies' own corpora.
Planned
впереди
Scaling learning from experience
The next growth axis is not model size but the volume of training on the model's own attempts and mistakes.
Branches
Milestones on the line
To the line