Reinforcement Learning Basics

A reward is the single number the environment hands you on one step. But a greedy agent that only chased the next reward would walk straight into traps. What you really care about is the return — the sum of all future rewards, with later ones discounted.

Read it left to right: take this step's reward in full, then add next step's reward shrunk by gamma, then the one after shrunk by gamma², and keep going. Because gamma is below 1, far-future rewards barely register — which also keeps the sum finite even if it never ends. The worked example below computes this for a real reward list and then does the one update step that turns an observed return into a learned value.

# The two numbers at the heart of RL: the return and the value
# Plain Python — no libraries needed.

# A reward is the single number the environment hands back each step.
# The RETURN G_t is the TOTAL future reward from a point in time,
# but with future rewards shrunk by a discount factor gamma (0..1).
#
#   G_t = r_t + gamma*r_{t+1} + gamma^2*r_{t+2} + ...
#
# gamma < 1 means "a reward now is worth more than the same reward later".

rewards = [1, 0, 2, 5]   # rewards collected over 4 time ste
...

A single return is what happened in one run. A value function is the return you expect on average. There are two of them, and the only difference is how much they pin down:

Q is the more useful of the two because it scores every action — so the agent can simply pick the action with the highest Q. And the two connect neatly: the value of a state is the value of its best action, written V(s) = max over actions Q(s, a). The example computes both for a small two-action state.

# V(s) vs Q(s,a): two flavours of "how good is this?"
#
#   V(s)    = expected return if you START in state s and act well after.
#   Q(s,a)  = expected return if you START in state s, take action a NOW,
#             then act well after.
#
# Q is more detailed: it scores each ACTION, so you can pick the best one.
# Relationship: V(s) = the Q of the best action = max over a of Q(s, a).

gamma = 0.9

def discounted_return(rewards):
    """Total reward in one episode, discounting later steps."""
 
...

The return G_t is an infinite sum, which sounds impossible to compute. The Bellman equation is the trick that makes it easy. It notices that the future, after one step, is just another value — so the whole sum folds into a tiny recursive rule:

In plain English: the value of where you are equals the reward you get now, plus the discounted value of wherever you land next (s′). You don't need to look at the entire future — you only need this step's reward and an estimate of the next state. That's the insight that turns RL from "sum over infinity" into a loop you can actually run.

RL problems come in two shapes, and they change how you think about the return:

This is the second reason discounting exists: for a continuing task the reward sum would run to infinity, but multiplying each future reward by gamma below 1 makes the total settle on a finite number. A terminal state (the end of an episode) is treated as having value 0 — there's no future left to discount.

A policy is the agent's strategy — its rule for choosing an action in each state. Algorithms differ in whether the policy they learn about is the same one they act with:

• On-policy — you learn about the very policy you're following, exploration and all. Honest but cautious; it can't easily reuse old data. (SARSA is the classic example.)
• Off-policy — you act with one policy (often a curious, exploratory one) but learn the value of a different policy (usually the best greedy one). More flexible, and it lets you reuse stored experience — at the cost of being harder to keep stable. (Q-learning is off-policy.)