Streams & Buffers

Streams let Node process data piece by piece as it arrives instead of loading it all into memory at once, and Buffers are the fixed-length chunks of raw binary data that streams move around.

Learn Streams & Buffers in our free Node.js course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a quick reference.

Part of the free Node.js course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

By the end of this lesson you'll know what a Buffer holds and how to convert it to and from text, you'll read a large file chunk by chunk with a readable stream, and you'll copy a file efficiently by piping one stream into another — all while keeping memory use flat.

What You'll Learn in This Lesson

1️⃣ Buffers: Chunks of Raw Bytes

Before streams, you need to understand what they carry. A Buffer is Node's container for raw binary data — a fixed-length sequence of bytes. Text, images, network packets: at the lowest level they're all just bytes, and a Buffer is how Node holds them.

• • Buffer.from('hello') — encodes a string into bytes (UTF-8 by default).
• • buf.toString() — decodes the bytes back into readable text.
• • buf.length — the number of bytes , which is not always the number of characters (an emoji takes several bytes).

Read the worked example below — every line is explained — then run it. Notice that printing the Buffer directly shows its bytes as hexadecimal numbers.

2️⃣ Readable Streams: Data Piece by Piece

A readable stream hands you a file (or a network response, or any source) in small chunks instead of all at once. You open one with fs.createReadStream(path) and then listen for two key events:

• • stream.on('data', fn) — fires once for every chunk as it arrives. Without an encoding the chunk is a Buffer; pass 'utf8' to get strings instead.
• • stream.on('end', fn) — fires once , after the final chunk, telling you the source is exhausted.
• • stream.on('error', fn) — fires if something goes wrong (like a missing file). Skip it and an error will crash your whole program.

Why bother? Because fs.readFile loads the entire file into memory before you see a single byte. A 2 GB log file would need 2 GB of RAM. A stream processes that same file in ~64 KB chunks, so memory use stays low and constant no matter how big the file gets.

Your turn. The program below works once you fill in the three blanks marked ___ . Follow the 👉 hints, then run it and compare with the expected output.

3️⃣ Writable Streams & .pipe()

A writable stream is the opposite of a readable one: it's a destination you push chunks into . You create one with fs.createWriteStream(path) . The magic happens when you connect a readable stream to a writable one with .pipe() :

• • source.pipe(dest) — forwards every chunk from source straight into dest , then closes the destination when the source ends.
• • It also handles backpressure for you: if the destination (a slow disk, say) can't keep up, pipe automatically pauses the source until it catches up.
• • The writable side emits 'finish' when every byte has been flushed.

This is how you copy a multi-gigabyte file using only a tiny, constant amount of memory — the whole copy is just source.pipe(dest) .

Mini-Challenge: Count a File's Bytes

No blanks this time — just a brief and an outline to keep you on track. Stream a file, total up the byte length of each chunk, and print the result. Write it yourself, run it, and check your output against the example in the comments.

Common Errors (and the fix)

• Reading a huge file with fs.readFileSync — this loads the entire file into memory at once, so a large file blows up your RAM (and Sync also freezes your whole program). Fix: use fs.createReadStream and process chunks as they arrive.
• Forgetting the 'end' event — if your final result depends on having seen the whole file, computing it inside 'data' gives wrong answers. Fix: accumulate in 'data' and report the result in 'end' , which fires once at the very end.
• Not handling the 'error' event — stream errors aren't thrown, they're emitted. An unhandled 'error' crashes the whole Node process. Fix: add stream.on('error', fn) to every stream.
• Buffer encoding mismatches — calling buf.toString('ascii') on UTF-8 bytes (or counting .length as characters) garbles non-English text and emoji. Fix: stick to 'utf8' for text and remember .length is bytes, not characters.
• Ignoring backpressure — writing to a destination faster than it can accept (looping dest.write() without checking its return value) balloons memory. Fix: use .pipe() , which manages backpressure for you automatically.

Pro Tips

• 💡 Prefer pipeline() over .pipe() in real apps: require('stream').pipeline wires streams together and forwards errors and cleanup for you, avoiding leaked file handles.
• 💡 Set the encoding once: passing 'utf8' to createReadStream means your 'data' chunks are strings, so you don't have to call .toString() on every one.
• 💡 Buffers are bytes, not characters: when measuring text size for the network, Buffer.byteLength(str) gives the true byte count that str.length won't.

📋 Quick Reference — Streams & Buffers

Concept

Node.js Syntax

Make a buffer

Buffer.from(str)

Buffer to string

buf.toString()

Open a read stream

fs.createReadStream(p)

Open a write stream

fs.createWriteStream(p)

Pipe one into another

src.pipe(dest)

Handle each chunk

stream.on('data', fn)

Detect the end

stream.on('end', fn)

Frequently Asked Questions

🎉 Lesson Complete!

• ✅ A Buffer holds a fixed-length chunk of raw bytes ; Buffer.from and .toString() convert to and from text
• ✅ A stream delivers data piece by piece instead of all at once
• ✅ fs.createReadStream + the 'data' and 'end' events read large files with low memory
• ✅ source.pipe(dest) copies streams efficiently and handles backpressure
• ✅ Always attach an 'error' handler — stream errors are emitted, not thrown
• ✅ Stream big files; only use readFile for small ones
• ✅ Next lesson: The http Module — build a web server using these same streams

Practice quiz

Continue this course

• Previous: Events & EventEmitter
• Next: The http Module