1989 Q-Learning
Reinforcement Learning
By Andy Oberlin, Founder, AImpact Nexus · Updated November 16, 2026

Listen to this chapter
Chris Watkins' PhD thesis introduced Q-learning, allowing AI agents to learn optimal actions through trial and error.
Why it matters
Enabled AI to learn from experience without explicit programming, foundational to modern game-playing AIs.
The story
Q-learning is a model-free reinforcement learning algorithm that has become one of the most important breakthroughs in the field. It allows an agent to learn to act optimally in a Markovian domain by experiencing the consequences of its actions, without requiring a model of the environment.
What was going on at the time
Q-learning was introduced by Christopher Watkins in his 1989 PhD thesis 'Learning from Delayed Rewards' at Cambridge University. It was a major advance in reinforcement learning, providing a simple and elegant way for an agent to learn to make optimal decisions in a wide range of environments. The algorithm has been used to solve a variety of problems, from playing games to controlling robots.
How it worked
Q-learning is based on the concept of a Q-function, which represents the expected utility of taking a given action in a given state and then following the optimal policy thereafter. The 'Q' in Q-learning stands for 'quality,' as the algorithm learns the quality of different actions in different states. The algorithm iteratively updates the Q-function based on the agent's experiences using the Bellman equation: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)], where s is the current state, a is the action taken, r is the reward received, s' is the new state, α is the learning rate, and γ is the discount factor. The algorithm is off-policy, meaning it can learn the optimal policy while following a different exploratory policy, making it very flexible and powerful.
Why it mattered
Q-learning's simplicity and effectiveness made it one of the most widely used reinforcement learning algorithms. It demonstrated that agents could learn complex behaviors through trial and error, without explicit programming. The algorithm has been applied to numerous real-world problems, from game playing to robotics control, demonstrating the broad applicability of reinforcement learning.
How people reacted
The introduction of Q-learning was recognized as a significant advance in reinforcement learning. Researchers appreciated its elegant formulation and its ability to learn optimal policies through experience. The algorithm's success helped to establish reinforcement learning as an important subfield of machine learning.
What it left behind
Q-learning is a cornerstone of modern reinforcement learning. It is the basis for many more advanced algorithms, including Deep Q-Networks (DQNs), which have been used to achieve superhuman performance in a variety of games. The algorithm is a testament to the power of reinforcement learning and its potential to create intelligent agents that can learn from experience. Q-learning has been particularly influential in game AI, robotics, and autonomous systems.
In their words
“Q-learning is a simple way for agents to learn how to act optimally in controlled Markovian domains.”
Timeline
- 1989 Christopher Watkins publishes 'Learning from Delayed Rewards' PhD thesis
- Early 1990s Q-learning gains adoption in reinforcement learning research
- 2000s Algorithm applied to robotics and autonomous systems
- 2013 DeepMind combines Q-learning with deep neural networks (DQN)
- 2015 DQN achieves human-level performance on Atari games
- 2016 Q-learning principles incorporated into AlphaGo
Key players
- Chris Watkins
- Peter Dayan
Did you know?
- Based on dynamic programming principles
- Powers modern game AI like AlphaGo
- The 'Q' stands for 'quality' of state-action pairs
Further reading
- Wikipedia: Q-learning
Technical details and applications
- Learning from Delayed Rewards
Christopher Watkins' original 1989 PhD thesis