Attention Is All You Need: The Translation Problem That Led to ChatGPT

The useful part

In the spring of 2017, Aidan Gomez was twenty years old, interning at Google, and sleeping in the office. At three in the morning, he and Ashish Vaswani were still working in a Google office. Their eight-person team had spent twelve weeks pushing toward a paper deadline, sometimes sleeping on couches behind a curtain patterned like neurons.

How it works

• It introduced the Transformer, the architecture behind GPT, BERT, and the line of work that eventually produced ChatGPT.
• Researchers suspected that the single note would become a bottleneck as sentences grew.
• The strange part is that the team found it while working on translation.
• Its newer neural system translated whole sentences instead of stitching together short phrases.
• The translations sounded more natural because the model could use more of the sentence as context.

What to take from it

The paper that came out of those late nights was called Attention Is All You Need. The architecture underneath it had a problem. Neural translation changed the job into one end-to-end model: read a sentence, build an internal representation, then generate it in another language.

Details worth keeping

They were trying to make machines better at translation. Google Translate had already made a large jump the year before. One influential 2014 system used two stacked LSTMs.