Q-Learning & Deep Q-Networks

By the end of this lesson you'll be able to build a Q-table, apply the Bellman update by hand, and write an epsilon-greedy agent that learns the best path through a grid world — all in plain Python.

Learn Q-Learning & Deep Q-Networks in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🌍 Real-World Analogy: Learning a Maze by Trial and Reward

Imagine you're dropped into a hedge maze with a chocolate bar hidden at the exit. You have no map. The first time, you wander randomly — left here, right there — until you stumble onto the exit and get the chocolate.

Each time you reach a junction (a state ) you remember which turn (an action ) eventually paid off. Over many runs you build a mental note for every junction: "from here, going right got me to the chocolate fastest." That mental note is exactly a Q-table — a score for every junction-and-turn pair.

Two instincts make you learn well. You discount faraway rewards (chocolate ten turns away is worth less than chocolate one turn away), and you sometimes explore a turn you've never tried, just in case there's a shortcut. Q-Learning is just this maze instinct written as a formula.

1 The Q-table — One Score Per (State, Action)

The heart of Q-Learning is the Q-table . Q(s,a) stores the agent's best guess of the total future reward it will earn if it takes action a in state s and then keeps acting optimally afterwards. "Q" stands for quality — the quality of that move.

You don't need a fancy data structure. A plain Python dictionary keyed by a (state, action) tuple works perfectly for small worlds. Every entry starts at 0.0 because the agent begins knowing nothing.

2 The Bellman Update — How Learning Actually Happens

Every time the agent takes an action and sees what happens, it nudges one cell of the Q-table using this rule:

Read it piece by piece — in plain English it just says "move my old guess a little closer to a better guess":

• r — the reward you just received for this move.
• maxQ(s',a') — the best Q-value available from the next state s' . This is the "what comes after" estimate.
• r + γ · maxQ(s',a') — the TD target : a fresher, better estimate of what Q(s,a) should be.
• [ target − Q(s,a) ] — the TD error : how wrong your old guess was.
• α · — take only a fraction of that correction, so one surprise doesn't overwrite everything.

It's called off-policy temporal-difference learning , but you don't need the jargon to use it. "Temporal difference" just means you learn from the gap between two estimates taken one step apart.

3 Learning Rate α and Discount Factor γ

How big a step you take toward the new estimate.

• α = 0: never learns (ignores new info).
• α small (0.1): slow, smooth, stable.
• α = 1: throws away the old value entirely each step — unstable.

How much you value future reward versus reward right now.

• γ near 0: short-sighted — grabs instant rewards.
• γ = 0.9–0.99: plans ahead (the usual choice).
• γ = 1: values distant reward fully — can fail to converge.

4 Epsilon-Greedy — Explore vs Exploit

There's a tension at the core of learning. If the agent always picks the action it currently thinks is best (pure greedy), it can lock onto a mediocre path and never discover a better one. But if it acts randomly forever, it never cashes in what it learned.

Epsilon-greedy is the simple fix: with probability ε (epsilon) act randomly ( explore ), otherwise pick the best known action ( exploit ).

5 Worked Example — One Update, Start to Finish

Here's the whole loop in miniature: a Q-table as a dictionary, a single Bellman update applied by hand, and an epsilon-greedy pick. Read every comment, run it, and watch Q(A,right) move from 0 to 5.

🎯 Your Turn #1 — Finish the Bellman Update

Fill in the two blanks so the update runs. The next state B already has learned values, so this time max_next won't be zero.

🎯 Your Turn #2 — Build Epsilon-Greedy

Complete the explore and exploit branches. Run it 1000 times and confirm the agent mostly picks the higher-value action.

🌍 When Tables Run Out: Deep Q-Networks

Everything above scales to any small world. But a Q-table needs one cell per state-action pair, and that breaks down fast: an Atari screen has more possible pixel states than atoms in the observable universe. You can't store a table that big, and you'd never visit each cell enough to learn it.

A Deep Q-Network (DQN) swaps the table for a neural network. Instead of looking up Q(s,a) , the network predicts it from the raw state, so similar states share what they've learned — the GPS that generalises instead of the notebook that memorises. DeepMind's 2013 DQN learned to play Atari games straight from pixels using exactly this idea, plus two stabilisers:

• Experience replay: store past (s, a, r, s') transitions and train on random batches, breaking correlation between consecutive steps.
• Target network: compute the TD target with a slowly-updated copy of the network, so you're not chasing a moving target.

The learning rule is still the Bellman update you mastered here — only the storage changed from a dict to a net.

6 Common Errors (And How to Fix Them)

These are the mistakes that quietly stop an agent from learning. None of them throw a Python error — the code runs, it just never gets good.

With alpha = 1.0 each update overwrites the old value with one noisy sample, so the Q-values jump around and never settle.

✅ Fix: use a small, steady α (0.1 is a safe default); lower it if values oscillate.

alpha = 0.001 barely moves the estimate each step, so training crawls and may not converge in the episodes you run.

✅ Fix: raise α (try 0.1) or run far more episodes. There's a trade-off — too high is unstable, too low is glacial.

gamma = 1.0 can stop the Q-values from converging at all; gamma = 0.1 makes the agent ignore the goal and chase tiny immediate rewards.

✅ Fix: stay in the usual 0.9–0.99 range so the agent plans ahead without diverging.

A pure-greedy agent commits to the first path it stumbles on and never discovers the better one — it gets stuck in a local optimum.

✅ Fix: keep epsilon > 0 so the agent keeps trying new actions while it learns.

Leaving epsilon = 0.5 for the whole run means half the agent's moves stay random even after it has learned the optimal path, capping its performance.

✅ Fix: start epsilon high and multiply it down toward a small floor (e.g. epsilon = max(0.05, epsilon * 0.99) ).

Trying to build a Q-table for chess or raw pixels means billions of cells — you run out of memory and never visit each state enough to learn it.

✅ Fix: when states can't be enumerated, switch from a table to a function approximator (a neural network) — that's a DQN.

📋 Quick Reference

Symbol / Term

Meaning

Typical value

Q(s,a)

Estimated total future reward for action a in state s

a number in the table

α (alpha)

Learning rate — step size toward the new estimate

0.1

γ (gamma)

Discount factor — weight on future reward

0.9 – 0.99

ε (epsilon)

Exploration rate — chance of a random action

1.0 → 0.05 (decayed)

Bellman update

Q(s,a) += α·[r + γ·maxQ(s',a') − Q(s,a)]

applied every step

DQN

Neural network replaces the Q-table for huge state spaces

when states are uncountable

❓ Frequently Asked Questions

🎯 Mini-Challenge: Train an Agent on a Line World

Now put it all together with the support removed. The starter below gives you only a comment outline — no filled-in logic. Build the training loop yourself: a Q-table dict, epsilon-greedy moves, and the Bellman update running for 500 episodes until every state learns to head toward the goal.

Lesson complete — you can teach an agent to learn!

You built a Q-table from a dictionary, applied the Bellman update by hand, tuned α and γ, and wrote an epsilon-greedy agent that explores then exploits. You also saw exactly when a table runs out and a Deep Q-Network takes over.

🚀 Up next: Policy Gradient Methods — instead of learning action values, learn the policy directly. These are the algorithms behind PPO and ChatGPT's RLHF training.

Practice quiz

Continue this course

• Previous: RL Basics
• Next: Policy Gradient Methods