Handling Missing Data (NaN, fillna, dropna)

Reviewed & published by Brayan K

Missing data in pandas is represented by NaN ("Not a Number"), and handling it means detecting those gaps and deciding whether to remove the affected rows or fill them with a sensible value — a step almost every real dataset requires before analysis.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Learn to detect NaN with isna()/notna(), count gaps, drop incomplete rows with dropna(), and fill holes with fillna() — including filling with a column's mean.

What You'll Learn in This Lesson

• What NaN is and why == can't detect it
• Detect gaps with df.isna() / df.isnull() and df.notna()
• Count missing values per column with df.isna().sum()
• Remove incomplete rows with df.dropna()
• Control dropna with axis, how, and subset
• Fill gaps with fillna(value), ffill, and a column mean

1 Detecting Missing Data — isna(), notna(), and counting

Pandas marks missing values with NaN. To create one in code you use numpy.nan. Because NaN == NaN is False, you can't find gaps with == — instead use df.isna() (identical to df.isnull()) which returns a True/False mask, and df.notna() for the opposite.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Alice", "Bob", "Carol"],
    "score": [90, np.nan, 70],
})

df.isna()             # True where the value is missing
df.notna()            # True where the value is present
df.isna().sum()       # count of NaN per column

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Alice", "Bob", "Carol", "Dan"],
    "score": [90, np.nan, 70, np.nan],
    "city":  ["NYC", "LA", np.nan, "LA"],
})

# Count missing values per column
print(df.isna().sum())

# A boolean mask of where data is present
print(df["score"].notna())
# 0     True
# 1    False
# 2     True
# 3    False
# Name: score, dtype: bool

# ✅ Expected output:
# name     0
# score    2
# city     1
# dtype: int64
# 0     True
# 1    False
# 2     True
# 3    False
# Name: score, dtype: bool

2 Removing Missing Data — df.dropna()

df.dropna() deletes rows that contain any NaN and returns a new DataFrame. Three arguments fine-tune it:

• axis — 0 (default) drops rows; 1 drops whole columns that contain NaN.
• how — "any" (default) drops if any value is NaN; "all" drops only if every value is NaN.
• subset — only consider NaN in the listed columns when deciding what to drop.

df.dropna()                       # drop any row with a NaN
df.dropna(how="all")              # drop only fully-empty rows
df.dropna(axis=1)                 # drop columns that contain NaN
df.dropna(subset=["score"])       # drop rows only if "score" is NaN

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Alice", "Bob", "Carol", "Dan"],
    "score": [90, np.nan, 70, np.nan],
    "city":  ["NYC", "LA", np.nan, "LA"],
})

# Default: drop every row that has ANY NaN
print(df.dropna())

# Only drop rows where "score" specifically is missing
print(df.dropna(subset=["score"]))

# ✅ Expected output:
#     name  score city
# 0  Alice   90.0  NYC
#     name  score city
# 0  Alice   90.0  NYC
# 2  Carol   70.0  NaN

3 Filling Missing Data — df.fillna()

Often you want to keep every row and replace gaps instead. df.fillna(value) substitutes a constant; method="ffill" forward-fills the last valid value down; and a very common technique is filling a numeric column with its own mean.

df.fillna(0)                                # every NaN -> 0
df.fillna(method="ffill")                   # carry the previous value down
df["score"].fillna(df["score"].mean())     # fill with the column mean

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Alice", "Bob", "Carol", "Dan"],
    "score": [90, np.nan, 70, np.nan],
})

# Fill the "score" gaps with the column's mean (mean ignores NaN)
mean_score = df["score"].mean()         # (90 + 70) / 2 = 80.0
df["score"] = df["score"].fillna(mean_score)
print(df)

# ✅ Expected output:
#     name  score
# 0  Alice   90.0
# 1    Bob   80.0
# 2  Carol   70.0
# 3    Dan   80.0

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "day":   ["Mon", "Tue", "Wed", "Thu"],
    "sales": [100, np.nan, np.nan, 160],
})

# 🎯 YOUR TURN:
# 1) Count how many sales values are missing.
# 2) Forward-fill the gaps so each NaN copies the value above it.
# Replace the ___ blanks.

print(df["sales"].___().sum())        # hint: detect missing -> count
df["sales"] = df["sales"].fillna(method=___)   # hint: "ffill" as a string

print(df)
# Expected output:
# 2
#    day  sales
# 0  Mon  100.0
# 1  Tue  100.0
# 2  Wed  100.0
# 3  Thu  160.0

! Common Pitfalls (And How to Fix Them)

❌ Comparing to NaN with ==

df[df["score"] == np.nan]   # ❌ always empty — NaN != NaN

df[df["score"].isna()]      # ✅ rows where score is missing

❌ Forgetting that dropna/fillna return copies

df.dropna()        # ❌ result discarded, df unchanged

✅ Fix: reassign the result:

df = df.dropna()   # ✅ keep the cleaned frame

🎯 Mini Challenge: Clean a Survey Table

A survey DataFrame has gaps. Clean it step by step.

• Print the per-column missing count
• Fill missing age with the column mean
• Drop any row still missing a rating

import pandas as pd
import numpy as np

survey = pd.DataFrame({
    "id":     [1, 2, 3, 4],
    "age":    [30, np.nan, 40, 50],
    "rating": [4, 5, np.nan, 3],
})

# 1) how many gaps per column?
print(survey.isna().sum())

# 2) fill age with its mean
survey["age"] = survey["age"].fillna(survey["age"].mean())

# 3) drop rows still missing a rating
survey = survey.dropna(subset=["rating"])
print(survey)

# ✅ Expected output:
# id        0
# age       1
# rating    1
# dtype: int64
#    id   age  rating
# 0   1  30.0     4.0
# 1   2  40.0     5.0
# 3   4  50.0     3.0

❓ Frequently Asked Questions

Lesson complete — your data is clean and analysis-ready!

You can detect NaN with isna(), tally gaps with isna().sum(), remove incomplete rows with dropna(), and fill holes with fillna() — including with a column's mean and forward-fill.

Practice quiz

Continue this course

• Previous: Adding, Modifying & Dropping Columns
• Next: Sorting and Ranking — Order data by values or index and assign ranks within columns
• Quick reference: Pandas cheat sheet › Cleaning & Missing Data
• From the blog: Data Preprocessing for Machine Learning

Frequently asked questions

What exactly is NaN in pandas?

NaN stands for 'Not a Number' and is the marker pandas uses for a missing value. It is technically a special float, which is why an integer column with a missing value is often promoted to float dtype.

Should I use dropna() or fillna()?

Use dropna() when missing rows are few and you can afford to discard them; use fillna() when you want to keep every row and replace gaps with a sensible value such as 0, a column mean, or the previous value via forward-fill.

Why can't I find NaN with == ?

NaN is never equal to anything, including itself, so df['x'] == NaN is always False. Use the dedicated methods df.isna() / df.isnull() to detect missing values and df.notna() for the present ones.