Performance: vectorization & memory

Reviewed & published by Brayan K

Performance tuning in pandas is the practice of making your code faster and leaner by replacing slow Python row-loops with vectorized operations that run in compiled C, and by shrinking memory with smaller dtypes and the category type — so the same analysis handles far larger datasets.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• Why vectorized operations beat loops and iterrows
• Replacing .apply with vectorized math and np.where
• Downcasting dtypes with astype and pd.to_numeric
• Using the category dtype for low-cardinality strings
• Measuring memory with df.memory_usage(deep=True)
• The idea behind %timeit benchmarking

1 Vectorization vs Loops

The single biggest pandas speed lesson: operate on whole columns at once, not row by row. A vectorized expression like df["a"] * df["b"] runs in compiled C over the entire array; a Python loop or iterrows() runs interpreted Python for every row and is typically 10–100× slower.

import pandas as pd

df = pd.DataFrame({
    "price": [10, 20, 30, 40],
    "qty":   [1, 2, 3, 4],
})

# 🐢 SLOW: explicit Python loop over rows (don't do this on big data)
totals_loop = []
for _, row in df.iterrows():
    totals_loop.append(row["price"] * row["qty"])
df["total_loop"] = totals_loop

# 🚀 FAST: one vectorized expression over the whole column
df["total_vec"] = df["price"] * df["qty"]

print(df)
# total_loop and total_vec are identical: [10, 40, 90, 160]
# But the vectorized version is dramatically faster at scale.

# ✅ Expected output:
#    price  qty  total_loop  total_vec
# 0     10    1          10         10
# 1     20    2          40         40
# 2     30    3          90         90
# 3     40    4         160        160

2 Avoid .apply When a Vectorized Op Exists

.apply is still a per-row Python call. For conditional logic, np.where (or .clip, .map, boolean masks) does the same job vectorized.

import pandas as pd
import numpy as np

df = pd.DataFrame({"score": [55, 72, 90, 41, 68]})

# 🐢 Slower: apply runs a Python function per row
df["grade_apply"] = df["score"].apply(lambda s: "Pass" if s >= 60 else "Fail")

# 🚀 Faster: np.where is fully vectorized
df["grade_vec"] = np.where(df["score"] >= 60, "Pass", "Fail")

print(df)
# grade_apply and grade_vec match exactly:
# ['Fail', 'Pass', 'Pass', 'Fail', 'Pass']

# Many string tasks are vectorized too, via the .str accessor:
names = pd.Series(["  Alice ", "BOB", "carol"])
print(names.str.strip().str.title().tolist())
# ['Alice', 'Bob', 'Carol']  -- no apply needed

# ✅ Expected output:
#    score grade_apply grade_vec
# 0     55        Fail      Fail
# 1     72        Pass      Pass
# 2     90        Pass      Pass
# 3     41        Fail      Fail
# 4     68        Pass      Pass
# ['Alice', 'Bob', 'Carol']

3 Shrinking Memory: dtypes & category

pandas defaults to wide dtypes (int64, float64, object strings). Downcasting numbers to int32/int16 and converting low-cardinality text columns to category can slash memory. Measure with df.memory_usage(deep=True).

import pandas as pd

df = pd.DataFrame({
    "id":     [1, 2, 3, 4],
    "city":   ["London", "London", "Paris", "London"],  # repeats -> low cardinality
    "visits": [10, 20, 30, 40],
})

before = df.memory_usage(deep=True)
print("Before:")
print(df.dtypes)

# Downcast integers and convert the repetitive string column
df["id"] = df["id"].astype("int16")
df["visits"] = pd.to_numeric(df["visits"], downcast="integer")
df["city"] = df["city"].astype("category")

after = df.memory_usage(deep=True)
print("After:")
print(df.dtypes)

# The numeric columns are fixed-width, so this saving is identical on every
# machine: int64 -> int16 and int8 is a 5x cut.
print("numeric bytes:", before[["id", "visits"]].sum(), "->", after[["id", "visits"]].sum())

# 'city' is left out of that figure on purpose, for two reasons worth knowing.
#
# 1. Both object AND category columns hold real Python str objects, and CPython
#    3.12 made every str 8 bytes smaller — so the totals differ by Python
#    version. Print before["city"] and after["city"] to see yours.
# 2. On FOUR rows, category barely helps: measured here it is 251 -> 237 bytes,
#    because the integer codes and the lookup table cost nearly as much as the
#    strings they replace. Category pays off at thousands of repeated values —
#    see the categorical lesson, where the same trick is a 40x cut — not at four.

# ✅ Expected output:
# Before:
# id         int64
# city      object
# visits     int64
# dtype: object
# After:
# id           int16
# city      category
# visits        int8
# dtype: object
# numeric bytes: 64 -> 12

🎯 Your Turn: Vectorize It

Replace the slow loop with a single vectorized expression that applies a 10% discount. Fill in the blank.

import pandas as pd

df = pd.DataFrame({"price": [100, 200, 300]})

# 🎯 YOUR TURN: compute a 10% discount with ONE vectorized line
df["sale_price"] = df["price"] * ___
print(df)

# EXPECTED OUTPUT:
#    price  sale_price
# 0    100        90.0
# 1    200       180.0
# 2    300       270.0

📋 Quick Reference

🏆 Mini Challenge: Optimise a DataFrame

Vectorize a calculation and shrink the frame's memory footprint, then confirm the savings.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "product_id": range(1, 6),
    "category":   ["A", "B", "A", "A", "B"],   # low cardinality
    "units":      [5, 0, 12, 3, 0],
    "price":      [2.0, 5.0, 1.5, 4.0, 3.0],
})

# 1) Vectorized revenue + a vectorized 'in_stock' flag
df["revenue"] = df["units"] * df["price"]
df["in_stock"] = np.where(df["units"] > 0, "Yes", "No")

# Measure AFTER adding the columns, not before. Comparing a 4-column frame with
# a 6-column one would only prove that six columns weigh more than four — the
# question here is what downcasting saves, so both sides need the same columns.
wide = df.memory_usage(deep=True).sum()

# 2) Shrink dtypes
df["product_id"] = df["product_id"].astype("int16")
df["category"] = df["category"].astype("category")
df["in_stock"] = df["in_stock"].astype("category")

lean = df.memory_usage(deep=True).sum()
print(df)

# Print wide and lean to see the real figures (880 -> 723 bytes here). They are
# not the claim: the category columns hold Python str objects, whose size
# changed in CPython 3.12, so your totals depend on your Python. The direction
# does not.
print("downcasting shrank the same frame:", lean < wide)

# ✅ Expected output:
#    product_id category  units  price  revenue in_stock
# 0           1        A      5    2.0     10.0      Yes
# 1           2        B      0    5.0      0.0       No
# 2           3        A     12    1.5     18.0      Yes
# 3           4        A      3    4.0     12.0      Yes
# 4           5        B      0    3.0      0.0       No
# downcasting shrank the same frame: True

❓ Frequently Asked Questions

Lesson complete — your pandas is now fast and lean!

You know to vectorize instead of loop, to skip .apply when a vectorized op exists, and to cut memory with smaller dtypes and category. These habits let you scale from toy data to millions of rows.

Practice quiz

Continue this course

• Previous: MultiIndex (Hierarchical Indexing)
• Next: The Index: set_index, reset_index & reindex — Work with the Index — set a meaningful index, reset it, and reindex for fast lookups
• Quick reference: Pandas cheat sheet

Frequently asked questions

Why is vectorization faster than a Python loop or iterrows?

Vectorized pandas/NumPy operations run as compiled C code over whole arrays at once, so the per-element Python overhead disappears. A for-loop or df.iterrows() executes interpreted Python for every single row and rebuilds objects each time, which is typically tens to hundreds of times slower. Whenever a column-wide operation exists, prefer it over looping.

When is .apply acceptable and when should I avoid it?

.apply runs a Python function per row or per element, so it is still essentially a loop and is slow on large data. Reach for it only when no vectorized equivalent exists. If you can express the logic with arithmetic, comparisons, .str methods, np.where, or .map, use those instead — they are vectorized and far faster.

How does the 'category' dtype save memory?

For a column with few distinct string values repeated many times (low cardinality), category stores each unique value once and represents the column as small integer codes pointing at that lookup table. This can shrink memory by an order of magnitude and also speeds up groupby and comparisons. Check the impact with df.memory_usage(deep=True).