Files Streams

Master efficient file handling, streaming patterns, and processing massive datasets that don't fit in memory.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

📊 What You'll Learn

This lesson teaches professional data handling techniques used in:

• Machine learning systems & data pipelines
• Analytics & ETL jobs
• Log processing & monitoring systems
• Web scrapers & API integrations
• Large-scale data transformations
• Stream files without loading them into memory
• Process multi-gigabyte datasets efficiently
• Build composable data pipelines
• Handle compressed and binary data
• Implement parallel file processing

📥 Python Download & Setup

Part 1: File I/O Fundamentals & Streaming Basics

1. Python's File I/O Model (Text vs Binary, Buffered vs Unbuffered)

• • Raw I/O = Receiving individual items one by one (slow)
• • Buffered I/O = Receiving full pallets at a time (efficient)
• • Text I/O = Unpacking and translating foreign labels into English

When you open a file, Python uses a layered I/O system:

Layer

What It Does

Speed

Raw I/O

Direct interaction with OS file descriptors

Slowest

Buffered I/O

Uses memory buffers (reads/writes in chunks)

Fast

Text I/O

Decodes bytes → Unicode strings

Automatic encoding

Modes:

• "w" → write (overwrite)
• "a" → append
• "b" → binary mode
• "rb" → read binary
• "wb" → write binary

Buffering matters: Python reads data in chunks internally to reduce syscalls, making it ideal for large datasets.

2. Reading Files the Right Way (Avoid .read() on Large Files)

This loads the entire file into memory, which is disastrous for large logs.

Approach

Memory Usage

For 10GB File

f.read()

Entire file in RAM

10GB+ RAM needed 💥

for line in f:

One line at a time

~KB of RAM ✓

Advantages:

• ✔ No RAM explosion
• ✔ Pause/resume possible
• ✔ Efficient buffering
• ✔ Ideal for multi-GB files

3. Using Buffered Streams Explicitly

For very large binary files (videos, models, audio):

4. Writing Files Efficiently (Avoid Tiny Writes)

5. Working With CSV Files at Scale

6. Handling JSON Without Crashing RAM

7. Working With Very Large Binary Files

8. Memory Mapping (mmap) — Fast Random Access to Huge Files

Method

Random Access Speed

f.seek() + f.read()

Medium

Low (read size)

f.read() entire file

Fast (after load)

File size 💥

mmap

Instant

Near zero

9. Chunk Processing for Massive Datasets

10. Using Iterators & Generators in Data Pipelines

11. Working With Compressed Data (ZIP, GZIP)

12. Handling Data That Doesn't Fit In RAM

13. Avoiding the Most Common File-I/O Mistakes

• ❌ Reading entire large files into memory
• ❌ Using .read() instead of iteration
• ❌ Building huge lists from file rows
• ❌ Tiny writes inside loops
• ❌ Forgetting to close files
• ❌ Failing to stream compressed data
• ❌ Loading massive JSON using json.load()
• ❌ Storing temporary files unnecessarily

14. Best Practices Recap

• ✔ Use with for everything
• ✔ Stream line-by-line
• ✔ Use chunked reads for large binaries
• ✔ Use ijson for large JSON
• ✔ Use csv reader instead of list(reader)
• ✔ Use memory mapping for random access
• ✔ Use generators to build pipelines
• ✔ Use compression modules without extracting

Part 2: Advanced Streaming & Dataset Engineering

1. Understanding Buffered I/O Depth

2. Random Access With Seek

3. Streaming API Data

4. Chunked CSV with Pandas

5. Producer/Consumer with Backpressure

6. Live Log Tailing

7. Temporary Files

8. Multiprocessing for Parallel Files

9. Binary Struct Parsing

10. Composable Pipeline Functions

11. Avoiding Common Streaming Pitfalls

• ❌ Forgetting .close() outside a with
• ❌ Inefficient small reads
• ❌ Building giant Python lists
• ❌ Using pandas on files too large
• ❌ Decompressing entire files unnecessarily
• ❌ Excessive text splits / regex in hot loops
• ❌ Ignoring buffer sizes
• ❌ Reading binary as text

Part 3: High-Volume File & Data Processing

1. Streaming Compressed Files

2. Zero-Copy with Memoryview

3. Async File I/O

4. Multiprocessing with imap

5. Rolling Window Processing

6. Endless Streaming Pipelines

7. Atomic File Writes

8. Common Pitfalls in Large-Dataset Work

• ❌ Naïve .read() on giant files
• ❌ Unbounded queues
• ❌ Excessive process spawning
• ❌ Forgetting to chunk operations
• ❌ Mixing CPU & I/O tasks in same thread
• ❌ Converting everything to pandas DataFrame
• ❌ Relying on recursion for streaming

🎓 Final Summary

You've now mastered professional file handling and large-scale data processing in Python.

• Stream massive files without loading them into memory
• Build efficient data pipelines with generators
• Process compressed and encrypted data on the fly
• Handle binary formats and structured records
• Use memory mapping for fast random access
• Implement multiprocessing for parallel file processing
• Work with columnar formats like Parquet
• Build real-time log monitoring systems
• Handle terabyte-scale datasets efficiently

📋 Quick Reference — Files & Streams

Syntax

What it does

with open(path, 'rb') as f:

Open file in binary read mode

pathlib.Path(path)

Modern path manipulation

for chunk in iter(f.read, b''):

Read large file in chunks

csv.DictReader(f)

Read CSV as list of dicts

io.StringIO(data)

In-memory file-like object

You can now handle files of any size efficiently, parse CSV/JSON/binary formats, and stream large datasets without memory issues.

Up next: Advanced Collections — master Counter, deque, ChainMap, itertools, and functools.

Practice quiz

Continue this course

• Previous: Packaging & PyPI
• Next: Collections