2017
Attention Is All You Need
The paper that defined the next decade.
Milestones on the line · ARCHITECTURE
Transformers
A universal network design that handles text, images, audio and code equally well — and keeps getting better when you simply make it bigger.
Attention gave us an architecture that scales almost without limit. The same block works for text, code, audio and video.
StatusPASSED
TypeMilestones on the line
Marker2017
What exists today7
Researched
2018
BERT
Language pre-training becomes the industry default.
2019
GPT-2
Coherent text generation stops being a toy.
2020
Vision Transformer
The same architecture works on images — the transformer turns out to be universal.
2021
Mixture of experts
Sparse models: more parameters without a proportional compute increase.
In progress
сейчас
Fighting the quadratic cost
Attention costs grow as the square of the input length. Linear and hybrid schemes are the main engineering work here.
Planned
впереди
A successor architecture
The design has held for nine years. Every major lab is looking for a replacement, but no alternative has yet won at scale.
Milestones on the line
To the line