1986 Backpropagation
Neural Network Training
By Andy Oberlin, Founder, AImpact Nexus · Updated November 16, 2026

Listen to this chapter
Rumelhart, Hinton, and Williams popularized backpropagation, enabling multi-layer neural networks to learn from data.
Why it matters
Provided the mathematical foundation for modern deep learning, though it took 20+ years to show its full potential.
The story
The 1986 paper by Rumelhart, Hinton, and Williams was a landmark in the history of neural networks. It popularized the backpropagation algorithm, a method for training multi-layer neural networks. While the algorithm had been independently discovered by others earlier, this paper was the first to demonstrate that it could be used to train deep neural networks to solve complex problems.
What was going on at the time
Backpropagation was a major breakthrough that made it possible to train deep neural networks. This led to a resurgence of interest in neural networks after years of decline following the first AI winter. The algorithm provided a practical method for training networks with hidden layers, overcoming a major limitation of earlier neural network models. The paper was published in Nature, one of the world's most prestigious scientific journals, which helped to establish its credibility.
How it worked
Backpropagation is a supervised learning algorithm that works by calculating the gradient of the loss function with respect to the weights of the network. This gradient is then used to update the weights in a process called gradient descent. The 'backpropagation' part of the name comes from the fact that the algorithm works by propagating the error signal from the output layer of the network back to the input layer. The algorithm uses the chain rule from calculus to efficiently compute gradients for all weights in the network. This makes it possible to train networks with multiple hidden layers, which was previously considered impractical.
Why it mattered
The 1986 paper demonstrated that backpropagation could be used to train networks to solve a variety of problems, including learning the past tense of English verbs and recognizing handwritten digits. These demonstrations helped to convince the AI community that neural networks were a viable approach to machine learning. The success of backpropagation helped to revive interest in neural networks and laid the foundation for the deep learning revolution of the 2010s.
How people reacted
The publication of the backpropagation paper generated significant excitement in the AI community. Researchers recognized that this could be the breakthrough needed to make neural networks practical for real-world applications. However, computational limitations meant that it would take another two decades before the full potential of backpropagation and deep learning could be realized.
What it left behind
Backpropagation is one of the most important algorithms in the history of AI. It is the workhorse of modern deep learning and has been used to achieve state-of-the-art results in a wide range of tasks, including image recognition, natural language processing, and speech recognition. The 1986 paper is one of the most cited papers in the field of computer science. Geoffrey Hinton, along with Yoshua Bengio and Yann LeCun, received the 2018 Turing Award for their work on deep learning, which built upon the foundation laid by the backpropagation algorithm.
In their words
“We describe a new learning procedure, back-propagation, for networks of neurone-like units.”
Timeline
- 1960s-1970s Backpropagation algorithm independently discovered by multiple researchers
- 1986 Rumelhart, Hinton, and Williams publish landmark paper in Nature
- 1986-2000s Algorithm slowly gains adoption despite computational limitations
- 2006 Hinton's deep learning breakthrough builds on backpropagation
- 2012 AlexNet uses backpropagation to achieve breakthrough in image recognition
- 2018 Hinton receives Turing Award for deep learning work
Key players
- David Rumelhart
- Geoffrey Hinton
- Ronald Williams
Did you know?
- The algorithm was actually discovered in the 1960s
- Required significant computing power to be practical
- Hinton would later win the 2018 Turing Award for this work
Further reading
- Wikipedia: Backpropagation
Technical details and history
- Nature: Learning representations by back-propagating errors
The original 1986 paper