Time Series Ops: shift, diff & pct_change

Reviewed & published by Brayan K

Time-series operations compare each row to an earlier or later one — lagging values with shift, measuring the absolute step with diff, and the relative step with pct_change — to reveal trends, returns, and momentum in ordered data.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You will lag and lead with shift(), compute period-over-period change with diff(), find returns with pct_change(), and accumulate with cumsum() and cummax() over a DatetimeIndex.

What You'll Learn in This Lesson

• Lag and lead values with shift(+1 / -1)
• Measure period-over-period change with diff()
• Compute returns and growth with pct_change()
• Accumulate with cumsum() and cummax()
• Why a sorted DatetimeIndex matters
• Read the NaN at the edges of a shift

1 shift() — Lag and Lead

shift(1) pushes every value down one row, so each row now carries the previous period's value — a lag. shift(-1) pulls values up, giving each row the next period's value — a lead. The row left empty at the top or bottom becomes NaN.

import pandas as pd

s = pd.Series([10, 12, 9, 15])

s.shift(1)     # lag: previous value
# 0     NaN
# 1    10.0
# 2    12.0
# 3     9.0

s.shift(-1)    # lead: next value
# 0    12.0
# 1     9.0
# 2    15.0
# 3     NaN

import pandas as pd

dates = pd.date_range("2024-01-01", periods=4, freq="D")
df = pd.DataFrame({"price": [10, 12, 9, 15]}, index=dates)

# Put yesterday's price next to today's
df["yesterday"] = df["price"].shift(1)
print(df)

# ✅ Expected output:
#             price  yesterday
# 2024-01-01     10        NaN
# 2024-01-02     12       10.0
# 2024-01-03      9       12.0
# 2024-01-04     15        9.0

2 diff() and pct_change() — Absolute vs Relative Change

diff() subtracts each value from the one before it, giving the absolute change in units. pct_change() divides that change by the previous value, giving the relative change — the return or growth rate. Both leave the first row as NaN because there is nothing earlier to compare with.

s = pd.Series([100, 120, 90])

s.diff()         # absolute step
# 0     NaN
# 1    20.0      (120 - 100)
# 2   -30.0      (90 - 120)

s.pct_change()   # relative step (a fraction)
# 0     NaN
# 1    0.20      (+20%)
# 2   -0.25      (-25%)

import pandas as pd

s = pd.Series([100, 120, 90, 99], name="revenue")

print(s.diff())

print(s.pct_change().round(3))

# ✅ Expected output:
# 0     NaN
# 1    20.0
# 2   -30.0
# 3     9.0
# Name: revenue, dtype: float64
# 0     NaN
# 1    0.20
# 2   -0.25
# 3    0.10
# Name: revenue, dtype: float64

3 cumsum() and cummax() — Running Accumulators

Cumulative methods carry a result forward down the column. cumsum() gives a running total — perfect for cumulative revenue. cummax() tracks the highest value seen so far, the basis for "record high to date" and drawdown analysis. There is also cummin() and cumprod().

s = pd.Series([3, 1, 4, 2])

s.cumsum()    # running total
# 3, 4, 8, 10

s.cummax()    # highest seen so far
# 3, 3, 4, 4

import pandas as pd

dates = pd.date_range("2024-03-01", periods=4, freq="D")
df = pd.DataFrame({"daily": [3, 1, 4, 2]}, index=dates)

df["running_total"] = df["daily"].cumsum()
df["best_so_far"]   = df["daily"].cummax()
print(df)

# ✅ Expected output:
#             daily  running_total  best_so_far
# 2024-03-01      3              3            3
# 2024-03-02      1              4            3
# 2024-03-03      4              8            4
# 2024-03-04      2             10            4

import pandas as pd

s = pd.Series([50, 55, 44, 66], name="stock")

# 🎯 YOUR TURN:
# Compute the day-over-day RETURN (relative change, a fraction).
# Replace ___ with the method that gives percentage change.

returns = s.___().round(3)
print(returns)

# Expected output:
# 0    NaN
# 1    0.1
# 2   -0.2
# 3    0.5
# Name: stock, dtype: float64

! Common Errors (And How to Fix Them)

❌ Computing change on unsorted dates

Out-of-order rows give nonsense diffs:

df["delta"] = df["price"].diff()   # ❌ if df is not sorted by date

✅ Fix: sort the index first:

df = df.sort_index()
df["delta"] = df["price"].diff()   # ✅

❌ Confusing diff with pct_change

s.diff()         # ❌ if you actually wanted a percentage return

✅ Fix: diff = units, pct_change = fraction:

s.pct_change()   # ✅ 0.20 means +20%

🎯 Mini Challenge: Build a Mini Stock Report

Turn a daily price column into a small analytics table.

• Add change = day-over-day diff
• Add return_pct = percentage change rounded to 3 dp
• Add peak = running maximum with cummax

import pandas as pd

dates = pd.date_range("2024-05-01", periods=4, freq="D")
df = pd.DataFrame({"close": [100, 110, 99, 120]}, index=dates)

df["change"] = df["close"].diff()
df["return_pct"] = df["close"].pct_change().round(3)
df["peak"] = df["close"].cummax()
print(df)

# ✅ Expected output:
#             close  change  return_pct  peak
# 2024-05-01    100     NaN         NaN   100
# 2024-05-02    110    10.0       0.100   110
# 2024-05-03     99   -11.0      -0.100   110
# 2024-05-04    120    21.0       0.212   120

❓ Frequently Asked Questions

Lesson complete — your data can compare across time!

You can lag and lead with shift, measure absolute change with diff, relative change with pct_change, and accumulate with cumsum and cummax on a sorted DatetimeIndex.

Practice quiz

Continue this course

• Previous: GroupBy transform & filter
• Next: Plotting with DataFrame.plot() — Make quick line, bar, histogram, and scatter charts straight from a DataFrame
• Quick reference: Pandas cheat sheet

Frequently asked questions

What does shift() actually do?

shift(1) moves every value down one row, so each row now shows the previous period's value — a lag. A negative argument like shift(-1) moves values up, giving you the next period's value — a lead. The vacated rows at the edge are filled with NaN because there is no earlier or later value to borrow.

How is diff() related to shift()?

diff() is shorthand for value minus its shifted self: df['x'] - df['x'].shift(1). It gives the period-over-period change. The first row is NaN because there is no prior value to subtract from. You can pass diff(7) to compare against 7 periods ago instead of 1.

When should I use pct_change() instead of diff()?

Use diff() for the absolute change in units (sales went up by 40) and pct_change() for the relative change as a fraction (sales went up by 0.25, i.e. 25%). pct_change is the standard way to compute returns in finance and growth rates in analytics.

Why does a DatetimeIndex matter for these operations?

shift, diff, and pct_change all work on row position, so they assume rows are already in time order. A sorted DatetimeIndex guarantees that ordering and makes the lag meaningful. It also unlocks time-aware tools like resampling and frequency-based shifting.