Pandas & Spark
Reviewed & published by Brayan K
Apache Spark is a distributed engine for processing data far too big for one machine, spreading the work across a cluster. With PySpark and the pandas API on Spark, you can use familiar DataFrame code at terabyte scale.
Part of the free Pandas course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Learn how Spark differs from pandas, the RDD vs DataFrame split, lazy transformations vs actions, reading data with spark.read, the pandas API on Spark and its Koalas history, and when Spark is worth the overhead.
What You'll Learn in This Lesson
- What Spark and PySpark are
- RDDs vs Spark DataFrames
- Lazy transformations vs actions
- Reading data with spark.read
- The pandas API on Spark and Koalas
- When Spark is the right tool
1 Spark, RDDs, and DataFrames
Where pandas runs on one machine, Spark spreads a job across a whole cluster. Its original abstraction is the RDD (Resilient Distributed Dataset) — a low-level distributed collection. On top of it sits the higher-level DataFrame API with a schema and the Catalyst optimizer, which is what you should reach for most of the time.
- RDD: low-level distributed objects, map/reduce style
- DataFrame: columnar, schema-aware, optimized — preferred
- PySpark: the Python API over both
The runnable example below uses plain pandas to compute a per-region total — the same answer a Spark DataFrame would compute across a cluster.
import pandas as pd
# Plain pandas equivalent of the Spark job below
sales = pd.DataFrame({
"region": ["North", "South", "North", "South", "North"],
"amount": [100, 80, 50, 90, 120],
})
print(sales.groupby("region")["amount"].sum())
# North -> 270, South -> 170
# ✅ Expected output:
# region
# North 270
# South 170
# Name: amount, dtype: int642 Lazy Transformations vs Actions
Spark is lazy. Transformations like filter, select, and groupBy only build a plan; nothing runs yet. Actions like collect, count, and show trigger execution. This lets Spark's Catalyst optimizer plan the entire job before a single byte moves.
3 The pandas API on Spark, Koalas, and When to Use Spark
If you love the pandas syntax, the pandas API on Spark (pyspark.pandas) lets you write pandas-style code that runs on Spark's distributed engine. It grew out of the Koalas project, a separate library that has since been merged directly into Apache Spark.
- Data is huge (hundreds of GB to TB) and distributed
- You already run on a cluster (e.g. Databricks, EMR, YARN)
- You need fault-tolerant, parallel processing at scale
🎯 Your Turn: The pandas Equivalent
Before distributing the job to Spark, prototype it in pandas. Fill in the blank to total the amount per region. Expected output: North 270, South 170.
import pandas as pd
df = pd.DataFrame({
"region": ["North", "South", "North", "South", "North"],
"amount": [100, 80, 50, 90, 120],
})
# 🎯 YOUR TURN: group by 'region' and total the 'amount'
totals = df.groupby("___")["amount"].sum()
print(totals)
# Expected output: North -> 270, South -> 170📋 Quick Reference
| Task | Code (PySpark) |
|---|---|
| Start session | SparkSession.builder.getOrCreate() |
| Read CSV | spark.read.csv("f.csv", header=True) |
| Transformation (lazy) | df.groupBy("k").sum() |
| Action (runs) | df.show() / df.collect() |
| pandas API on Spark | import pyspark.pandas as ps |
❓ Frequently Asked Questions
Lesson complete — you can think at cluster scale!
You now understand what Spark and PySpark are, the difference between RDDs and DataFrames, lazy transformations vs actions, how spark.read loads data, the pandas API on Spark and its Koalas history, and when Spark is worth its overhead.
Practice quiz
What is Apache Spark primarily designed for?
- Distributed big-data processing across a cluster
- Single-machine plotting
- Editing images
- Compiling C code
Answer: Distributed big-data processing across a cluster. Spark is a distributed engine that processes huge datasets across many machines in a cluster.
What is the Python API for Spark called?
- SparkSQL only
- SparkPy
- PySpark
- PyArrow
Answer: PySpark. PySpark is the official Python API for Apache Spark.
Which is the lower-level, older Spark abstraction?
- Dataset
- RDD (Resilient Distributed Dataset)
- Series
- DataFrame
Answer: RDD (Resilient Distributed Dataset). The RDD is Spark's original low-level distributed collection; DataFrames are the higher-level API on top.
In Spark, which of these is a TRANSFORMATION (lazy)?
- collect()
- show()
- count()
- filter()
Answer: filter(). filter is a lazy transformation; it just records intent until an action runs.
Which of these is an ACTION that triggers execution?
- collect()
- select()
- filter()
- withColumn()
Answer: collect(). collect() is an action: it forces Spark to run the lazy plan and return results.
How do you read a CSV in PySpark?
- SparkContext.csv('path')
- spark.read.csv('path', header=True)
- pd.read_csv('path')
- spark.open('path')
Answer: spark.read.csv('path', header=True). spark.read.csv(...) on a SparkSession reads files into a Spark DataFrame.
What does the pandas API on Spark (pyspark.pandas) give you?
- A way to plot maps
- A SQL-only interface
- Local pandas with no Spark
- A pandas-like API that runs on Spark's distributed engine
Answer: A pandas-like API that runs on Spark's distributed engine. pyspark.pandas (import pyspark.pandas as ps) offers a pandas-like API backed by Spark.
What was Koalas, historically?
- A Spark scheduler
- A plotting tool
- The earlier standalone library that brought the pandas API to Spark
- A file format
Answer: The earlier standalone library that brought the pandas API to Spark. Koalas was a separate project that added a pandas API on Spark; it was merged into Spark as pyspark.pandas.
When does Spark make the most sense?
- For tiny CSVs on a laptop
- For terabyte-scale data across a cluster
- For drawing charts
- Never with structured data
Answer: For terabyte-scale data across a cluster. Spark's overhead pays off at very large, distributed, cluster-scale workloads, not small local data.
Why is Spark's laziness useful?
- It lets the optimizer (Catalyst) plan the whole job before running
- It avoids reading data forever
- It disables the cluster
- It never computes anything
Answer: It lets the optimizer (Catalyst) plan the whole job before running. Deferring work until an action lets Spark's Catalyst optimizer plan and optimize the entire job.
Continue this course
- Previous: Polars: A Faster DataFrame
- Next: Geospatial Data with GeoPandas — Work with geometries, CRS, spatial joins, and maps using GeoDataFrames
- Quick reference: Pandas cheat sheet
Frequently asked questions
What is the difference between an RDD and a Spark DataFrame?
The RDD (Resilient Distributed Dataset) is Spark's original low-level abstraction: a distributed collection of objects you manipulate with map and reduce. The DataFrame is a higher-level, columnar API built on top, with a schema and the Catalyst optimizer. For most work, the DataFrame API is preferred because it is faster and easier.
What is the difference between a transformation and an action in Spark?
Transformations such as filter, select, and groupBy are lazy: they just describe what to do and build a plan. Actions such as collect, count, and show trigger execution, forcing Spark to run the whole plan across the cluster and return or save results.
What is the pandas API on Spark, and how does it relate to Koalas?
The pandas API on Spark, imported as pyspark.pandas, lets you write familiar pandas-style code that actually runs on Spark's distributed engine. It grew out of the Koalas project, which was a separate library that has since been merged directly into Apache Spark.