Checkpoint: Cleaning & Wrangling

Reviewed & published by Brayan K

A checkpoint is a hands-on review where you combine every cleaning skill — dtypes, dropping, renaming, counting, querying, and replacing — to wrangle one messy DataFrame into shape.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You'll recap the whole cleaning toolkit, tackle a multi-step build challenge with a full solution, and test yourself with a checkpoint quiz before moving on.

What You've Learned in This Unit

• The Index — set_index, reset_index, reindex
• Data types — dtypes, astype, to_numeric
• Categorical data — category dtype & ordered ranks
• Dropping rows & deduplicating with drop_duplicates
• Renaming columns & the index labels
• Counting with value_counts, unique, nunique
• Readable filtering with query() and eval()
• Replacing values with replace, where, mask, clip

1 The Mission — One Messy DataFrame

Here is a small but realistically grubby export from a sign-up form. The numbers arrived as text, there is a duplicate row, the headers are ugly, a status code needs expanding, and there is an under-age entry. Your job is to turn it into clean, analysis-ready data using the skills from this unit.

import pandas as pd

raw = pd.DataFrame({
    "User Name": ["Ann", "Bob", "Ann", "Cy", "Dee"],
    "Age":       ["30", "15", "30", "41", "22"],   # text, not numbers!
    "Plan":      ["P", "F", "P", "X", "F"],         # codes to expand
    "City":      ["NYC", "LA", "NYC", "NYC", "SF"],
})

🛠️ Build Challenge: Clean It Up, Step by Step

Work through these five steps. Each one uses a skill from this unit.

• Tidy the headers: lowercase and replace spaces with underscores
• Convert age from text to integers
• Drop the duplicate user row
• Expand the plan codes (P→pro, F→free, X→trial) with replace
• Keep only adults (18+) using query

import pandas as pd

raw = pd.DataFrame({
    "User Name": ["Ann", "Bob", "Ann", "Cy", "Dee"],
    "Age":       ["30", "15", "30", "41", "22"],
    "Plan":      ["P", "F", "P", "X", "F"],
    "City":      ["NYC", "LA", "NYC", "NYC", "SF"],
})

df = raw.copy()

# 1) tidy the headers
df.columns = df.columns.str.lower().str.replace(" ", "_")

# 2) age -> integers (fill in the rest!)
df["age"] = df["age"].astype("int")

# 3) drop the duplicate user row
# 4) expand plan codes with replace({...})
# 5) keep only adults with df.query(...)

print(df)
# Build the remaining steps, then compare with the solution below.

# ✅ Expected output:
#   user_name  age plan city
# 0       Ann   30    P  NYC
# 1       Bob   15    F   LA
# 2       Ann   30    P  NYC
# 3        Cy   41    X  NYC
# 4       Dee   22    F   SF

import pandas as pd

raw = pd.DataFrame({
    "User Name": ["Ann", "Bob", "Ann", "Cy", "Dee"],
    "Age":       ["30", "15", "30", "41", "22"],
    "Plan":      ["P", "F", "P", "X", "F"],
    "City":      ["NYC", "LA", "NYC", "NYC", "SF"],
})

df = raw.copy()

# 1) tidy the headers
df.columns = df.columns.str.lower().str.replace(" ", "_")

# 2) age -> integers
df["age"] = df["age"].astype("int")

# 3) drop the duplicate user row (Ann appears twice)
df = df.drop_duplicates()

# 4) expand the plan codes
df["plan"] = df["plan"].replace({"P": "pro", "F": "free", "X": "trial"})

# 5) keep only adults
df = df.query("age >= 18")

print(df)
# Final result:
#   user_name  age   plan city
# 0       Ann   30    pro  NYC
# 3        Cy   41  trial  NYC
# 4       Dee   22   free   SF

print(df["plan"].value_counts().to_dict())
# {'pro': 1, 'trial': 1, 'free': 1}

2 Verify Your Result

A clean pipeline is repeatable. Run the finished version and confirm the dtypes are right, the duplicate is gone, and only adults remain. Checking df.dtypes and len(df) at the end is a habit worth keeping.

import pandas as pd

raw = pd.DataFrame({
    "User Name": ["Ann", "Bob", "Ann", "Cy", "Dee"],
    "Age":       ["30", "15", "30", "41", "22"],
    "Plan":      ["P", "F", "P", "X", "F"],
    "City":      ["NYC", "LA", "NYC", "NYC", "SF"],
})

df = (raw
      .rename(columns=lambda c: c.lower().replace(" ", "_"))
      .assign(age=lambda d: d["age"].astype("int"))
      .drop_duplicates()
      .replace({"P": "pro", "F": "free", "X": "trial"})
      .query("age >= 18"))

print(df.dtypes["age"])
# int64

print("rows:", len(df))

# ✅ Expected output:
# int64
# rows: 3

import pandas as pd

df = pd.DataFrame({"age": ["20", "9", "33"], "name": ["A", "B", "C"]})

# 🎯 YOUR TURN:
# The age column is TEXT. Convert it to integers so the next line works,
# then the adult filter will behave. Fill in the dtype name (in quotes).

df["age"] = df["age"].astype("___")
print(df.query("age >= 18"))

# Expected output:
#   age name
# 0  20    A
# 2  33    C

! Common Errors (And How to Fix Them)

❌ Numeric filter matches nothing

Filtering a text column with a number compares strings, not values:

df.query("age >= 18")   # ❌ age is "30" (text) -> no matches

✅ Fix: convert the type first:

df["age"] = df["age"].astype("int")
df.query("age >= 18")   # ✅

❌ Cleaning steps quietly lost

Each step returns a new frame; without reassignment they vanish:

df.drop_duplicates()
df.replace({"P": "pro"})   # ❌ neither change is saved

✅ Fix: reassign or chain:

df = df.drop_duplicates().replace({"P": "pro"})   # ✅

📝 Checkpoint Quiz

Answer each in your head, then expand to check.

df.set_index("col"). Use reset_index() to push it back to an ordinary column.

At least one value is non-numeric text, so pandas stored the column as strings. Clean it and convert with astype, or use pd.to_numeric(col, errors="coerce") to turn bad values into NaN.

Sort by the timestamp, then df.drop_duplicates(subset=["email"], keep="last"). The subset judges duplicates by email; keep="last" retains the most recent.

Proportions that sum to 1 instead of raw counts — each category's share of the column. Multiply by 100 for percentages.

Prefix it with @: df.query("age > @cutoff"). Without the @, pandas looks for a column called cutoff.

where keeps the original value where cond is True and inserts 0 where it is False. mask is the opposite: it inserts 0 where cond is True. Mask hides (replaces) the True cells.

❓ Frequently Asked Questions

Checkpoint cleared — you can wrangle messy data!

You combined dtype fixes, deduplication, renaming, code expansion, and filtering into one repeatable pipeline — the everyday work of real data cleaning.

Practice quiz

Continue this course

• Previous: Replacing Values: replace, where & mask
• Next: Binning Continuous Data: cut & qcut — Group numbers into ranges with cut for fixed edges and qcut for quantiles
• Quick reference: Pandas cheat sheet

Frequently asked questions

What order should I clean a messy DataFrame in?

A reliable order is: fix dtypes first (so comparisons and math behave), then drop duplicates and unwanted rows, rename and tidy labels, replace or coerce bad values, and filter last. Doing dtypes early prevents subtle bugs — for example, '30' as text never satisfies a numeric > 18 test until it is converted.

Why did my numeric filter return nothing on cleaned-looking data?

Almost always because the column is still text (object dtype). The string '30' is not greater than the number 18, so the comparison silently matches nothing. Run df.dtypes, convert the column with astype or pd.to_numeric, then filter again.

Should I chain all my cleaning steps or do them one by one?

Both work. Beginners benefit from one step per line with a print() after each so they can see the effect. As you gain confidence, method chaining (df.dropna().drop_duplicates().rename(...)) reads cleanly and avoids intermediate variables. Either way, assign the final result back to a variable.