GroupBy transform & filter

Reviewed & published by Brayan K

GroupBy transform broadcasts a per-group statistic back to every original row so you can add it as a column, while GroupBy filter keeps or discards whole groups based on a test — two tools that go beyond the usual collapse-to-a-summary aggregation.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You will use groupby().transform() to add group means and z-scores, groupby().filter() to drop small or low-volume groups, and see exactly how both differ from .agg().

What You'll Learn in This Lesson

• How transform keeps the original shape
• Add a group mean back as a new column
• Compute a within-group z-score
• How filter keeps or drops whole groups
• Write a group predicate with len() or sum()
• Contrast transform and filter with agg

1 transform() — Same Shape, Group Stat Broadcast

A normal .agg() collapses each group to one row. .transform() does the opposite: it computes the group statistic, then spreads that one value back across every row in the group. The result has the same length and index as the input, so it slots straight into a new column.

import pandas as pd

df = pd.DataFrame({"team": ["A", "A", "B", "B"],
                   "pts":  [10, 20, 5, 15]})

# agg -> one row per group (shape shrinks)
df.groupby("team")["pts"].mean()
# A    15.0
# B    10.0

# transform -> one value per ORIGINAL row (shape kept)
df.groupby("team")["pts"].transform("mean")
# 0    15.0
# 1    15.0
# 2    10.0
# 3    10.0

import pandas as pd

df = pd.DataFrame({"team": ["A", "A", "B", "B", "B"],
                   "pts":  [10, 20, 5, 15, 25]})

# Add each team's average as a column next to the raw points
df["team_mean"] = df.groupby("team")["pts"].transform("mean")
print(df)

# ✅ Expected output:
#   team  pts  team_mean
# 0    A   10       15.0
# 1    A   20       15.0
# 2    B    5       15.0
# 3    B   15       15.0
# 4    B   25       15.0

2 A Within-Group Z-Score

Because transform keeps shapes aligned, you can build per-row standardised scores. A z-score measures how far a value sits from its group's mean, in units of the group's standard deviation. Pass a small custom function that returns a Series the same length as each group.

def zscore(x):
    return (x - x.mean()) / x.std()

df["z"] = df.groupby("team")["pts"].transform(zscore)
# each value standardised WITHIN its own team

import pandas as pd

df = pd.DataFrame({"team": ["A", "A", "B", "B"],
                   "pts":  [10, 30, 40, 60]})

def zscore(x):
    return (x - x.mean()) / x.std()

df["z"] = df.groupby("team")["pts"].transform(zscore)
print(df)

# ✅ Expected output:
#   team  pts         z
# 0    A   10 -0.707107
# 1    A   30  0.707107
# 2    B   40 -0.707107
# 3    B   60  0.707107

3 filter() — Keep or Drop Whole Groups

groupby().filter(func) hands each group to your function as a sub-DataFrame. If the function returns True, every row of that group survives; if False, the whole group is dropped. It is row selection at the group level — useful for removing rare categories or low-volume segments.

df = pd.DataFrame({"team": ["A", "A", "B", "C", "C", "C"],
                   "pts":  [1, 2, 9, 3, 4, 5]})

# keep only teams with more than 2 rows
df.groupby("team").filter(lambda g: len(g) > 2)
# only team C survives (it has 3 rows)

import pandas as pd

df = pd.DataFrame({"team": ["A", "A", "B", "C", "C", "C"],
                   "pts":  [1, 2, 9, 3, 4, 5]})

# Keep only teams that have more than 2 members
big = df.groupby("team").filter(lambda g: len(g) > 2)
print(big)

# ✅ Expected output:
#   team  pts
# 3    C    3
# 4    C    4
# 5    C    5

import pandas as pd

df = pd.DataFrame({"store": ["x", "x", "y", "y", "z"],
                   "sales": [100, 300, 50, 50, 800]})

# 🎯 YOUR TURN:
# Keep only stores whose TOTAL sales exceed 200.
# Replace ___ with the method that keeps/drops whole groups.

result = df.groupby("store").___(lambda g: g["sales"].sum() > 200)
print(result)

# Expected output:
#   store  sales
# 0     x    100
# 1     x    300
# 4     z    800

! Common Errors (And How to Fix Them)

❌ Using agg when you wanted a column

agg shrinks the frame, so this misaligns when assigned back:

df["m"] = df.groupby("team")["pts"].mean()   # ❌ wrong length / NaNs

✅ Fix: transform keeps the original shape:

df["m"] = df.groupby("team")["pts"].transform("mean")   # ✅

❌ filter function returns per-row booleans

df.groupby("team").filter(lambda g: g["pts"] > 5)   # ❌ returns a Series

✅ Fix: return ONE boolean per group:

df.groupby("team").filter(lambda g: g["pts"].mean() > 5)   # ✅

🎯 Mini Challenge: Normalise Then Prune

Combine both tools on one sales table.

• Add a region_mean column with transform
• Add a vs_mean column = sales minus that mean
• Keep only regions whose total sales exceed 150

import pandas as pd

df = pd.DataFrame({"region": ["N", "N", "S", "E", "E"],
                   "sales":  [100, 120, 80, 60, 110]})

df["region_mean"] = df.groupby("region")["sales"].transform("mean")
df["vs_mean"] = df["sales"] - df["region_mean"]

big = df.groupby("region").filter(lambda g: g["sales"].sum() > 150)
print(big)

# ✅ Expected output:
#   region  sales  region_mean  vs_mean
# 0      N    100        110.0    -10.0
# 1      N    120        110.0     10.0
# 3      E     60         85.0    -25.0
# 4      E    110         85.0     25.0

❓ Frequently Asked Questions

Lesson complete — group stats now ride along!

You can broadcast a group statistic onto every row with transform, build within-group z-scores, and keep or drop whole groups with filter — all without collapsing your data the way agg does.

Practice quiz

Continue this course

• Previous: Rolling & Expanding Windows
• Next: Time Series Ops: shift, diff & pct_change — Lag, difference, and compute returns over time with shift, diff, and pct_change
• Quick reference: Pandas cheat sheet

Frequently asked questions

How is transform different from agg?

agg collapses each group to a single summary row, so the result is shorter than the input. transform keeps the original shape — it computes a group statistic and then broadcasts it back to every row in that group, so the result lines up perfectly for assigning as a new column.

When would I use transform instead of agg plus merge?

Whenever you want a group statistic alongside the raw rows — like each row's value minus its group mean, or a group z-score. transform does in one step what would otherwise need agg followed by a merge back onto the original frame, and it preserves the index automatically.

What does groupby().filter() return?

filter keeps or drops entire groups based on a predicate that returns True or False for each group. The output is a subset of the original rows — every row of the surviving groups, and none of the rows of the rejected ones. It never changes individual values, only which groups remain.

Can the function passed to filter look at group size or sums?

Yes. The function receives each group as a sub-DataFrame, so you can test len(g) > 2, g['sales'].sum() > 1000, or any condition you like. Return a single boolean and pandas keeps the group when it is True.