2017 Transformers
Attention Is All You Need
By Andy Oberlin, Founder, AImpact Nexus · Updated November 16, 2026

Listen to this chapter
Google researchers introduced the Transformer architecture, revolutionizing natural language processing with self-attention mechanisms.
Why it matters
Became the foundation for GPT, BERT, and all modern large language models, transforming NLP forever.
The story
The Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need,' was a paradigm shift in natural language processing. It replaced the recurrent neural networks (RNNs) that had been the state of the art for sequence-to-sequence tasks with a new architecture based entirely on the attention mechanism. This new architecture was more parallelizable and required less time to train, while achieving better results.
What was going on at the time
The Transformer architecture revolutionized natural language processing by introducing a new way of processing sequential data that was both more efficient and more effective than previous approaches. Published at NeurIPS (Neural Information Processing Systems) in 2017, the paper was authored by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin from Google. It has become the foundation for virtually all modern large language models, including GPT, BERT, and their successors.
How it worked
The key innovation of the Transformer architecture is the self-attention mechanism. This mechanism allows the model to weigh the importance of different words in the input sequence when processing a given word, in contrast to RNNs which process the input sequence one word at a time. Key components include: Self-Attention Mechanism (allows the model to attend to different positions in the input sequence when encoding a particular position), Multi-Head Attention (uses multiple attention mechanisms in parallel, allowing the model to attend to information from different representation subspaces), Positional Encodings (added to give the model information about the order of words in the sequence, since the Transformer doesn't process sequences in order like RNNs), Feed-Forward Networks (each position is processed by a feed-forward network independently and identically), and Encoder-Decoder Structure (the original Transformer used an encoder-decoder architecture for machine translation, though later models like GPT use only the decoder and BERT uses only the encoder).
Why it mattered
The Transformer architecture has had a revolutionary impact on the field of natural language processing. It has become the standard architecture for a wide range of NLP tasks, including machine translation, text summarization, question answering, text generation, and sentiment analysis. The architecture is also the basis for many of the most successful large language models, including GPT-3, GPT-4, BERT, and T5. Beyond NLP, Transformers have been adapted for computer vision (Vision Transformers) and even protein folding (AlphaFold).
How people reacted
The publication of 'Attention Is All You Need' generated immediate excitement in the NLP community. Researchers recognized that this could fundamentally change how sequence-to-sequence problems were solved. The paper's clear presentation and strong empirical results helped it gain rapid adoption across the field.
What it left behind
The Transformer architecture is one of the most important inventions in the history of AI. It has transformed the field of natural language processing and has been a major driver of the recent progress in AI. The 'Attention Is All You Need' paper is one of the most influential papers in the history of computer science, with tens of thousands of citations. The success of Transformers has led to their application in domains beyond NLP, including computer vision and scientific computing, demonstrating the broad applicability of the attention mechanism.
What this means for your business today
Every AI tool your team uses today, from ChatGPT and Claude to Gemini and Copilot, runs on the Transformer. Its trick, weighing every word against every other word at once, is why AI can read a 40-page contract or a month of shift notes and answer questions about it. For a business, that means your own documents, email and ERP data can become something your team simply asks. The model is no longer the hard part. Connecting your data privately and safely, so the answers come from your business and stay in it, is.
In their words
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
Timeline
- 2017 Paper 'Attention Is All You Need' published at NeurIPS
- 2018 BERT (Bidirectional Encoder Representations from Transformers) released
- 2018 GPT (Generative Pre-trained Transformer) released
- 2019 GPT-2 demonstrates scaling benefits of Transformers
- 2020 GPT-3 shows emergent capabilities with 175B parameters
- 2020 Vision Transformers (ViT) apply architecture to computer vision
- Present Transformers dominate AI research across all domains
Key players
- Ashish Vaswani
- Noam Shazeer
- Niki Parmar
- Jakob Uszkoreit
Did you know?
- All eight authors are listed as equal contributors, in random order
- The "T" in GPT and ChatGPT stands for Transformer
- By 2023 every one of the eight authors had left Google, several to found AI companies
Questions people ask
What is a Transformer in AI?
A Transformer is a neural network design, introduced by Google researchers in the 2017 paper 'Attention Is All You Need', that reads a whole passage at once and uses 'attention' to weigh how every word relates to every other word. Because it processes text in parallel instead of word by word, it can be trained on far more data. It is the 'T' in GPT.
Why did the Transformer replace older approaches?
Earlier language models (recurrent networks such as LSTMs) read text one word at a time, which was slow to train and lost track of context over long passages. Attention lets a Transformer connect words that are far apart directly, and its parallel design runs well on modern GPUs, so models could grow much larger.
Which AI tools use Transformers today?
Essentially all of the major ones: ChatGPT and the GPT models, Claude, Gemini, Llama, and Microsoft Copilot are all built on the Transformer design, as are many image, speech and code models.
Further reading
- Wikipedia: Transformer (machine learning model)
Complete technical overview and history
- Attention Is All You Need
The original 2017 paper