Generators

Generators and iterators are two of Python's most powerful features — allowing you to process massive datasets, stream data, write memory-efficient code, build pipelines, and design systems that behave like professional-grade libraries.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• • What iterators are and how the iterator protocol works
• • How to create generators with yield
• • The difference between return and yield
• • Generator expressions and memory-efficient data pipelines
• • Real-world use cases: streaming large files, infinite sequences, lazy evaluation

What You'll Learn

This lesson will take you from advanced fundamentals → deep internal mechanics → real-world patterns used in production.

🔥 1. What Exactly Are Iterators?

An iterator is any object that can give you items one at a time. It follows a simple contract:

Method

What It Does

When It's Called

__iter__()

Returns the iterator itself

When you start a for-loop

__next__()

Returns the next item in sequence

Each iteration of the loop

StopIteration

Signals "no more items"

When sequence is exhausted

⚙️ 2. Why Iterators Matter

Without Iterators

With Iterators

Load entire 5GB file into RAM

Process one line at a time

Computer crashes with "Out of Memory"

Works smoothly with constant memory

Must wait for all data before starting

Start processing immediately

• Streaming: Process data as it arrives (like Netflix streaming)
• Pipelines: Chain operations together (like assembly lines)
• Lazy evaluation: Only compute when needed (saves CPU & RAM)
• Infinite sequences: Generate data forever without memory limits

⚡ 3. Enter Generators — The Shortcut to Iterators

A generator is Python's elegant shortcut for creating iterators using the yield keyword.

Iterator Class

Generator Function

~15 lines of code

~5 lines of code

Define __init__, __iter__, __next__

Just use yield

Manual state management

Automatic state saving

Instead of writing an entire class, just use yield :

This produces the same behavior as the iterator class — but with 90% less code.

🧠 4. How yield Works Internally

When Python sees yield , something magical happens:

Step 1: Function becomes a generator object (not executed yet!)

Step 4: Next next() resumes from exact pause point

Step 5: Repeat until function ends → StopIteration

return

yield

Function ends permanently

Function pauses, can resume

Returns single value

Can yield many values over time

All local variables lost

All local variables preserved

🧵 5. Real-World Example — Log File Streaming

Imagine parsing a 5GB log file . Without generators, you'd crash. With generators:

• ✔ Doesn't load the entire file
• ✔ Works line by line
• ✔ Streams infinitely large files

Python itself uses this pattern in its own IO libraries.

💧 6. Infinite Generators (Perfect for Simulations & AI)

• Reinforcement learning training loops
• Randomized datasets
• Unique ID generation

📦 7. Generator Pipelines (Functional Programming Style)

Chain generators together like Unix shell pipes ( command1 | command2 | command3 ):

Each step processes data lazily. Nothing loads into memory at once.

🧩 8. yield from — Delegating to Sub-Generators

yield from lets you delegate iteration to another generator:

This is cleaner and faster than nested loops.

🌀 9. Two-Way Generators (send() Method)

You can send values INTO generators, making them interactive:

• Async frameworks
• Stream processors
• Complex event simulation

⚙️ 10. Generator-Based Coroutines (Pre-asyncio Style)

Generators can act as coroutines — functions that can pause and receive data:

Now superseded by async def, but still heavily used in internal libraries and advanced scheduling systems.

📚 11. Generators as Context Managers

This pattern merges generators + cleanup logic.

🚀 12. Generator Expressions (Faster, Cleaner, Memory Efficient)

Structure

Memory Use

List comprehension

Loads entire list

Generator expression

Streams items one by one

🔬 13. Building Your Own Iterable Class (Advanced)

Here's a complete example with detailed comments:

🕹 14. How Python's For-Loops Use Iterators Internally

Understanding this gives you total control over how objects behave.

📊 15. Real-World Use Cases (Professional Level)

Streaming large CSVs with chunksize parameter

Generators still power internal scheduling logic.

🧪 16. Mini Project — Build a Streaming Data Pipeline

• A data reader
• A filter stage
• A transformer

This simulates how Airflow, Spark, and Pandas internally process data.

🎉 Conclusion

✔ How to use generators for memory-efficient processing

✔ How advanced frameworks use iterators under the hood

📋 Quick Reference — Generators

Syntax

What it does

🏆 Lesson Complete!

You understand lazy evaluation and how to build memory-efficient pipelines with generators — the same technique used inside Pandas, Airflow, and Spark.

Up next: Advanced Async & Await — write non-blocking concurrent code with asyncio.

Practice quiz

Continue this course

• Previous: Context Managers & the with Statement
• Next: Advanced Async & Await Patterns