The AI Timeline
Expert Systems Era

1986 Backpropagation

Neural Network Training

By Andy Oberlin, Founder, AImpact Nexus · Updated November 16, 2026

Backpropagation (1986)

Listen to this chapter

Rumelhart, Hinton, and Williams popularized backpropagation, enabling multi-layer neural networks to learn from data.

Why it matters

Provided the mathematical foundation for modern deep learning, though it took 20+ years to show its full potential.

The story

The 1986 paper by Rumelhart, Hinton, and Williams was a landmark in the history of neural networks. It popularized the backpropagation algorithm, a method for training multi-layer neural networks. While the algorithm had been independently discovered by others earlier, this paper was the first to demonstrate that it could be used to train deep neural networks to solve complex problems.

What was going on at the time

Backpropagation was a major breakthrough that made it possible to train deep neural networks. This led to a resurgence of interest in neural networks after years of decline following the first AI winter. The algorithm provided a practical method for training networks with hidden layers, overcoming a major limitation of earlier neural network models. The paper was published in Nature, one of the world's most prestigious scientific journals, which helped to establish its credibility.

How it worked

Backpropagation is a supervised learning algorithm that works by calculating the gradient of the loss function with respect to the weights of the network. This gradient is then used to update the weights in a process called gradient descent. The 'backpropagation' part of the name comes from the fact that the algorithm works by propagating the error signal from the output layer of the network back to the input layer. The algorithm uses the chain rule from calculus to efficiently compute gradients for all weights in the network. This makes it possible to train networks with multiple hidden layers, which was previously considered impractical.

Why it mattered

The 1986 paper demonstrated that backpropagation could be used to train networks to solve a variety of problems, including learning the past tense of English verbs and recognizing handwritten digits. These demonstrations helped to convince the AI community that neural networks were a viable approach to machine learning. The success of backpropagation helped to revive interest in neural networks and laid the foundation for the deep learning revolution of the 2010s.

How people reacted

The publication of the backpropagation paper generated significant excitement in the AI community. Researchers recognized that this could be the breakthrough needed to make neural networks practical for real-world applications. However, computational limitations meant that it would take another two decades before the full potential of backpropagation and deep learning could be realized.

What it left behind

Backpropagation is one of the most important algorithms in the history of AI. It is the workhorse of modern deep learning and has been used to achieve state-of-the-art results in a wide range of tasks, including image recognition, natural language processing, and speech recognition. The 1986 paper is one of the most cited papers in the field of computer science. Geoffrey Hinton, along with Yoshua Bengio and Yann LeCun, received the 2018 Turing Award for their work on deep learning, which built upon the foundation laid by the backpropagation algorithm.

In their words

“We describe a new learning procedure, back-propagation, for networks of neurone-like units.”
Rumelhart, Hinton, and Williams, From the 1986 Nature paper

Timeline

  1. 1960s-1970s Backpropagation algorithm independently discovered by multiple researchers
  2. 1986 Rumelhart, Hinton, and Williams publish landmark paper in Nature
  3. 1986-2000s Algorithm slowly gains adoption despite computational limitations
  4. 2006 Hinton's deep learning breakthrough builds on backpropagation
  5. 2012 AlexNet uses backpropagation to achieve breakthrough in image recognition
  6. 2018 Hinton receives Turing Award for deep learning work

Key players

  • David Rumelhart
  • Geoffrey Hinton
  • Ronald Williams

Did you know?

  • The algorithm was actually discovered in the 1960s
  • Required significant computing power to be practical
  • Hinton would later win the 2018 Turing Award for this work

Further reading