Efficient Formats: Parquet, Feather & Pickle

Reviewed & published by Brayan K

Parquet, Feather, and Pickle are binary file formats that store a DataFrame far faster and smaller than CSV while preserving exact dtypes, so a saved file loads back as the same data you saved.

Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You'll learn why columnar formats beat CSV, how to use to_parquet/read_parquet, and the always-available to_pickle/read_pickle pair.

What You'll Learn in This Lesson

• Why CSV is slow and loses dtypes
• How columnar Parquet stores data
• to_parquet() and read_parquet() syntax
• Where Feather fits vs Parquet
• Round-trip with to_pickle / read_pickle
• Pick the right format for the job

1 Why CSV Falls Short — Columnar to the Rescue

CSV is row-oriented plain text with no type information: every number is stored as characters and re-parsed on every read, and a date saved as text comes back as a string. Columnar binary formats like Parquet store each column together with its dtype and compression, so they read back the exact same data — much faster and far smaller.

# CSV: human-readable but typeless and uncompressed
#   "when" round-trips as a string, not a datetime

# Parquet: columnar, compressed, dtype-aware
df.to_parquet("data.parquet")          # write (needs pyarrow)
again = pd.read_parquet("data.parquet")  # read -> dtypes preserved

2 Parquet and Feather in Practice

The API mirrors every other pandas reader/writer. Parquet and Feather both need the pyarrow engine installed (pip install pyarrow). Parquet compresses hardest and is the archival standard; Feather (Arrow IPC) is tuned for the fastest possible read/write of intermediate results.

# --- Parquet ---
df.to_parquet("sales.parquet", compression="snappy")
pd.read_parquet("sales.parquet")
pd.read_parquet("sales.parquet", columns=["city", "pop"])  # only 2 cols

# --- Feather ---
df.to_feather("sales.feather")
pd.read_feather("sales.feather")

import pandas as pd
import io

df = pd.DataFrame({
    "event":  ["login", "logout", "purchase"],
    "when":   pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"]),
    "amount": [0.0, 0.0, 19.99],
})

# CSV loses the datetime dtype: "when" comes back as plain text
csv_text = df.to_csv(index=False)
from_csv = pd.read_csv(io.StringIO(csv_text))
print("via CSV:")
print(from_csv.dtypes)

# A binary format (pickle here) preserves it exactly
buf = io.BytesIO()
df.to_pickle(buf); buf.seek(0)
from_bin = pd.read_pickle(buf)
print("via pickle:")
print(from_bin.dtypes)

# ✅ Expected output:
# via CSV:
# event      object
# when       object
# amount    float64
# dtype: object
# via pickle:
# event             object
# when      datetime64[ns]
# amount           float64
# dtype: object

3 Pickle — The Always-Available Round-Trip

to_pickle/read_pickle serialise any DataFrame to a binary blob using Python's built-in pickle. No extra packages, perfect dtype fidelity, and it handles odd column types other formats reject. The trade-off: it is Python-only and unsafe to load from untrusted sources.

df.to_pickle("cache.pkl")     # write to disk
pd.read_pickle("cache.pkl")   # read it back, identical

# (the demos here use io.BytesIO so nothing touches your disk)

import pandas as pd
import io

df = pd.DataFrame({
    "city":    ["NYC", "LA", "SF"],
    "pop":     [8400000, 3900000, 880000],
    "coastal": [True, True, True],
})

# Write to a binary buffer, then read it straight back
buf = io.BytesIO()
df.to_pickle(buf)
buf.seek(0)
loaded = pd.read_pickle(buf)

print(loaded)

print("frames identical:", df.equals(loaded))

# ✅ Expected output:
#   city      pop  coastal
# 0  NYC  8400000     True
# 1   LA  3900000     True
# 2   SF   880000     True
# frames identical: True

import pandas as pd
import io

df = pd.DataFrame({"item": ["A", "B"], "qty": [3, 7]})
buf = io.BytesIO()

# 🎯 YOUR TURN:
# Save df to the buffer, rewind it, and read it back.
# Fill in the two method names (write, then read).

df.____(buf)        # serialise to the buffer
buf.seek(0)         # rewind so we can read from the start
print(pd.____(buf)) # read it back

# Expected output:
#   item  qty
# 0    A    3
# 1    B    7

! Common Errors (And How to Fix Them)

❌ "Missing optional dependency 'pyarrow'"

Parquet and Feather need the Arrow engine:

df.to_parquet("out.parquet")   # ❌ ImportError if pyarrow isn't installed

✅ Fix: install it, or fall back to pickle:

# pip install pyarrow
# ...or use the built-in format:
df.to_pickle("out.pkl")        # ✅ always available

❌ Reading a BytesIO buffer without rewinding

buf = io.BytesIO(); df.to_pickle(buf)
pd.read_pickle(buf)   # ❌ reads from the end -> EOFError / empty

✅ Fix: seek back to the start first:

buf.seek(0)
pd.read_pickle(buf)   # ✅

🎯 Mini Challenge: Lossless Cache

Prove a round-trip is lossless across mixed dtypes.

• Build a DataFrame with int, datetime, and bool columns
• Pickle it to a buffer and read it back
• Confirm it is identical and the datetime dtype survived

import pandas as pd
import io

df = pd.DataFrame({
    "id":     [1, 2, 3],
    "ts":     pd.to_datetime(["2024-05-01", "2024-05-02", "2024-05-03"]),
    "active": [True, False, True],
})

buf = io.BytesIO()
df.to_pickle(buf)
buf.seek(0)
back = pd.read_pickle(buf)

print(back)

print("identical:", df.equals(back))
print("ts dtype:", back["ts"].dtype)

# ✅ Expected output:
#    id         ts  active
# 0   1 2024-05-01    True
# 1   2 2024-05-02   False
# 2   3 2024-05-03    True
# identical: True
# ts dtype: datetime64[ns]

❓ Frequently Asked Questions

Lesson complete — you can store data the fast way!

You know why columnar Parquet and Feather beat CSV on speed, size, and dtype fidelity, and you can always fall back to the built-in to_pickle/read_pickle for a lossless round-trip.

Practice quiz

Continue this course

• Previous: Reading & Writing SQL
• Next: Ranking & Top-N (rank, nlargest) — Rank rows, find the top N, and rank within groups
• Quick reference: Pandas cheat sheet

Frequently asked questions

Why is Parquet faster and smaller than CSV?

Parquet is a columnar binary format: it stores each column together, compresses each one with a codec suited to its type, and records the schema and dtypes. CSV is row-oriented plain text with no types, so every value is re-parsed on read and there is no compression. On wide datasets Parquet files are commonly several times smaller and load far faster.

What is the difference between Parquet and Feather?

Both are columnar binary formats built on Apache Arrow and both preserve dtypes. Parquet compresses harder and is the standard for long-term storage and data lakes; Feather (Arrow IPC) is optimised for very fast read/write of intermediate results between processes. Use Parquet to archive, Feather to hand data around quickly.

Do I need to install anything to use Parquet or Feather?

Yes — both rely on the pyarrow library (pip install pyarrow). Pickle, by contrast, is built into Python and pandas, so to_pickle/read_pickle always work without extra dependencies.

Is Pickle safe to share with other people?

Not really. Pickle is a Python-only format that can execute arbitrary code when loaded, so you should only unpickle files you trust and that you created. For sharing data across tools, languages, or untrusted sources, prefer Parquet or CSV.