Q-Learning & Deep Q-Networks

The heart of Q-Learning is the Q-table. Q(s,a) stores the agent's best guess of the total future reward it will earn if it takes action a in state s and then keeps acting optimally afterwards. "Q" stands for quality — the quality of that move.

You don't need a fancy data structure. A plain Python dictionary keyed by a (state, action) tuple works perfectly for small worlds. Every entry starts at 0.0 because the agent begins knowing nothing.

# A Q-table is just a dictionary
Q = {}
states  = ["A", "B"]
actions = ["left", "right"]

for s in states:
    for a in actions:
        Q[(s, a)] = 0.0     # start every estimate at zero

print(Q[("A", "left")])     # 0.0  — no knowledge yet

# To act, read the row for the current state and pick the best action:
best = max(actions, key=lambda a: Q[("A", a)])
print(best)                 # 'left' (ties broken by first seen)

Every time the agent takes an action and sees what happens, it nudges one cell of the Q-table using this rule:

Read it piece by piece — in plain English it just says "move my old guess a little closer to a better guess":

• r — the reward you just received for this move.
• maxQ(s',a') — the best Q-value available from the next state s'. This is the "what comes after" estimate.
• r + γ · maxQ(s',a') — the TD target: a fresher, better estimate of what Q(s,a) should be.
• [ target − Q(s,a) ] — the TD error: how wrong your old guess was.
• α · — take only a fraction of that correction, so one surprise doesn't overwrite everything.

It's called off-policy temporal-difference learning, but you don't need the jargon to use it. "Temporal difference" just means you learn from the gap between two estimates taken one step apart.

# Same situation, two different gammas:
reward = 0
max_next = 100          # a big payoff is reachable from the next state

short_sighted = reward + 0.1 * max_next   # 10.0  — barely cares about the future
far_sighted   = reward + 0.95 * max_next  # 95.0  — strongly pulled toward the payoff

print(short_sighted)    # 10.0
print(far_sighted)      # 95.0

There's a tension at the core of learning. If the agent always picks the action it currently thinks is best (pure greedy), it can lock onto a mediocre path and never discover a better one. But if it acts randomly forever, it never cashes in what it learned.

Epsilon-greedy is the simple fix: with probability ε (epsilon) act randomly (explore), otherwise pick the best known action (exploit).

import random

epsilon = 0.1   # explore 10% of the time

def choose(state):
    if random.random() < epsilon:
        return random.choice(actions)              # EXPLORE: try something new
    return max(actions, key=lambda a: Q[(state, a)])  # EXPLOIT: pick the best known

# Early on, you want a HIGH epsilon (lots of exploration).
# As Q gets accurate, DECAY epsilon so the agent exploits what it learned:
epsilon = max(0.05, epsilon * 0.99)   # shrink each episode, but never below 0.05

Here's the whole loop in miniature: a Q-table as a dictionary, a single Bellman update applied by hand, and an epsilon-greedy pick. Read every comment, run it, and watch Q(A,right) move from 0 to 5.

# Q-LEARNING BY HAND — one Bellman update, no libraries
# The Q-table is just a dictionary: Q[(state, action)] = expected future reward.

import random
random.seed(0)

# A tiny world. The agent is in state "A" and can go "left" or "right".
# Start every estimate at 0.0 — the agent knows nothing yet.
Q = {
    ("A", "left"):  0.0,
    ("A", "right"): 0.0,
    ("B", "left"):  0.0,
    ("B", "right"): 0.0,
}

alpha = 0.5    # learning rate: how big a step we take toward the new estimate
gamma = 0.9
...

These are the mistakes that quietly stop an agent from learning. None of them throw a Python error — the code runs, it just never gets good.