Dropping & Deduplicating Rows

Reviewed & published by Brayan K

Dropping is how you remove rows you don't want from a DataFrame — either by label with df.drop(index=...) or by collapsing repeated records with drop_duplicates().

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Learn to delete rows by index, remove them by condition, flag repeats with duplicated(), and deduplicate by a business key with subset= and keep=.

What You'll Learn in This Lesson

• Delete rows by label with df.drop(index=...)
• Remove rows by condition with a boolean mask
• Drop a column quickly with drop(columns=...)
• Flag repeats with df.duplicated()
• Collapse duplicates with drop_duplicates()
• Control which copy stays with subset= and keep=

1 Dropping Rows — by Label and by Condition

df.drop(index=[...]) removes rows by their index labels. It is direct when you know exactly which rows to delete. To remove a column instead, use df.drop(columns=[...]) — same method, different axis. But for removing rows that meet a condition, the pandas idiom is to keep the rows you want with a boolean mask rather than naming the ones to drop.

import pandas as pd

df = pd.DataFrame({"name": ["A", "B", "C"], "age": [25, 17, 40]})

df.drop(index=[1])              # remove the row labelled 1
df.drop(columns=["age"])        # remove a whole column

# remove by condition = keep the opposite
df[df["age"] >= 18]            # keep adults; minors fall away

import pandas as pd

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Carol", "Dan"],
    "age":  [25, 17, 40, 16],
})

# Drop a specific row by its index label
print(df.drop(index=[1]))
#     name  age
# 0  Alice   25
# 2  Carol   40
# 3    Dan   16

# Drop by CONDITION = keep the rows you want
print(df[df["age"] >= 18])

# ✅ Expected output:
#     name  age
# 0  Alice   25
# 2  Carol   40
# 3    Dan   16
#     name  age
# 0  Alice   25
# 2  Carol   40

2 Finding Repeats — df.duplicated()

Before deleting anything, it pays to look. df.duplicated() returns a boolean Series: True for every row that is a repeat of an earlier one, False for the first time a row is seen. Sum it to count duplicates, or use it as a mask to view exactly which rows are about to be removed.

df.duplicated()                 # True where the WHOLE row repeats
df.duplicated().sum()           # how many duplicate rows exist
df[df.duplicated()]             # show the offending rows

df.duplicated(subset=["email"]) # judge repeats by email only

import pandas as pd

df = pd.DataFrame({
    "email": ["a@x.com", "b@x.com", "a@x.com", "c@x.com"],
    "name":  ["Ann", "Bob", "Ann", "Cy"],
})

# Flag whole-row repeats
print(df.duplicated().tolist())

print("dupes:", df.duplicated().sum())
# dupes: 1

# Inspect the repeat row before deleting it
print(df[df.duplicated()])
#      email name
# 2  a@x.com  Ann

# ✅ Expected output:
# [False, False, True, False]
# dupes: 1
#      email name
# 2  a@x.com  Ann

3 Removing Repeats — drop_duplicates(subset=, keep=)

drop_duplicates() deletes the repeated rows and hands back the cleaned table. Two arguments give you precise control: subset= chooses which columns define a duplicate (e.g. dedupe by email alone), and keep= picks which copy survives — "first" (default), "last", or False to discard every row that has any duplicate.

df.drop_duplicates()                       # exact-row duplicates, keep first
df.drop_duplicates(subset=["email"])       # one row per email, keep first
df.drop_duplicates(subset=["email"], keep="last")  # keep the newest
df.drop_duplicates(keep=False)             # drop ALL rows that repeat

import pandas as pd

df = pd.DataFrame({
    "email":   ["a@x.com", "a@x.com", "b@x.com", "b@x.com"],
    "updated": [1, 2, 1, 2],
})

# One row per email, keeping the most recently updated
clean = (df.sort_values("updated")
           .drop_duplicates(subset=["email"], keep="last"))
print(clean)

print("rows kept:", len(clean))
# rows kept: 2

# ✅ Expected output:
#      email  updated
# 1  a@x.com        2
# 3  b@x.com        2
# rows kept: 2

import pandas as pd

df = pd.DataFrame({
    "user": ["u1", "u2", "u1", "u3", "u2"],
    "plan": ["pro", "free", "pro", "pro", "free"],
})

# 🎯 YOUR TURN:
# Keep only ONE row per user (the first one seen).
# Fill in the argument that judges duplicates by the "user" column.

result = df.drop_duplicates(___=["user"])
print(result)

# Expected output:
#   user  plan
# 0   u1   pro
# 1   u2  free
# 3   u3   pro

! Common Errors (And How to Fix Them)

❌ "['age'] not found in axis"

Passing a column name as if it were a row label:

df.drop("age")   # ❌ pandas looks for a ROW labelled "age"

✅ Fix: say which axis you mean:

df.drop(columns=["age"])   # ✅ removes the column

❌ Duplicates "still there" after drop_duplicates

The result was never assigned back:

df.drop_duplicates()   # ❌ result discarded, df unchanged

✅ Fix: reassign (or use inplace=True):

df = df.drop_duplicates()   # ✅

🎯 Mini Challenge: Clean a Sign-up List

A messy registration export needs three cleaning passes.

• Drop rows where age is under 18
• Keep one row per email
• Report how many rows survived

import pandas as pd

df = pd.DataFrame({
    "email": ["a@x.com", "b@x.com", "a@x.com", "c@x.com"],
    "age":   [30, 15, 30, 22],
})

# 1) drop minors (keep adults)
df = df[df["age"] >= 18]

# 2) one row per email
df = df.drop_duplicates(subset=["email"])

print(df)
#      email  age
# 0  a@x.com   30
# 3  c@x.com   22

print("survivors:", len(df))

# ✅ Expected output:
#      email  age
# 0  a@x.com   30
# 3  c@x.com   22
# survivors: 2

❓ Frequently Asked Questions

Lesson complete — your tables are tidy!

You can drop rows by label or condition, drop columns with drop(columns=...), flag repeats with duplicated(), and collapse them with drop_duplicates(subset=, keep=).

Practice quiz

Continue this course

• Previous: Categorical Data
• Next: Renaming & Relabelling — Rename columns and index labels cleanly with rename, add_prefix, and add_suffix
• Quick reference: Pandas cheat sheet › Cleaning & Missing Data

Frequently asked questions

Should I use df.drop() or a boolean mask to remove rows?

Use df.drop(index=...) when you know the exact labels to remove. Use a boolean mask — df[df['x'] > 0] — when you are removing rows that meet a condition. The mask approach keeps the rows you want rather than naming the ones you don't, which is usually clearer and the standard pandas idiom for conditional removal.

What does the keep argument in drop_duplicates do?

keep decides which copy of a duplicate to retain. keep='first' (the default) keeps the earliest occurrence, keep='last' keeps the latest, and keep=False drops every row that has any duplicate, leaving only the truly unique rows. Choose based on whether the newest or oldest record is the one you trust.

What is the difference between duplicated() and drop_duplicates()?

duplicated() returns a boolean Series flagging which rows are repeats, so you can inspect or count them before deleting anything. drop_duplicates() actually removes those rows and returns the cleaned DataFrame. A common workflow is to look with duplicated() first, then remove with drop_duplicates().

Why use subset= with drop_duplicates?

By default a row counts as a duplicate only if EVERY column matches. subset=['email'] tells pandas to judge duplicates by that column alone, so two rows with the same email are collapsed even if other columns differ. This is how you deduplicate by a business key like customer ID or email.