Streams & Buffers

Reviewed & published by Brayan K

Streams let Node process data piece by piece as it arrives instead of loading it all into memory at once, and Buffers are the fixed-length chunks of raw binary data that streams move around.

Part of the free Node.js course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

By the end of this lesson you'll know what a Buffer holds and how to convert it to and from text, you'll read a large file chunk by chunk with a readable stream, and you'll copy a file efficiently by piping one stream into another — all while keeping memory use flat.

What You'll Learn in This Lesson

• What a Buffer is and why it stores raw bytes, not characters
• How to convert between Buffers and strings with Buffer.from and .toString()
• How to read a large file piece by piece with a readable stream
• Why streaming beats readFile for big or unknown-size files
• How to write data and copy files efficiently with .pipe()
• Why every stream needs an 'error' handler and what backpressure means

1️⃣ Buffers: Chunks of Raw Bytes

Before streams, you need to understand what they carry. A Buffer is Node's container for raw binary data — a fixed-length sequence of bytes. Text, images, network packets: at the lowest level they're all just bytes, and a Buffer is how Node holds them.

• • Buffer.from('hello') — encodes a string into bytes (UTF-8 by default).
• • buf.toString() — decodes the bytes back into readable text.
• • buf.length — the number of bytes, which is not always the number of characters (an emoji takes several bytes).

Read the worked example below — every line is explained — then run it. Notice that printing the Buffer directly shows its bytes as hexadecimal numbers.

// A Buffer is a fixed-length chunk of raw binary data.
// Buffer.from(string) encodes the text into bytes (UTF-8 by default).
const buf = Buffer.from('hello');

// .toString() decodes those bytes back into text.
console.log(buf.toString());   // -> hello

// .length is the number of BYTES, not characters.
console.log(buf.length);       // -> 5

// Printing the buffer itself shows the raw bytes in hex.
console.log(buf);              // -> <Buffer 68 65 6c 6c 6f>

// You can read an individual byte by index (68 = 'h').
console.log(buf[0]);          // -> 104  (0x68 in decimal)

2️⃣ Readable Streams: Data Piece by Piece

A readable stream hands you a file (or a network response, or any source) in small chunks instead of all at once. You open one with fs.createReadStream(path) and then listen for two key events:

• • stream.on('data', fn) — fires once for every chunk as it arrives. Without an encoding the chunk is a Buffer; pass 'utf8' to get strings instead.
• • stream.on('end', fn) — fires once, after the final chunk, telling you the source is exhausted.
• • stream.on('error', fn) — fires if something goes wrong (like a missing file). Skip it and an error will crash your whole program.

Why bother? Because fs.readFile loads the entire file into memory before you see a single byte. A 2 GB log file would need 2 GB of RAM. A stream processes that same file in ~64 KB chunks, so memory use stays low and constant no matter how big the file gets.

const fs = require('fs');

// The file this example reads. In your own project it would already exist;
// the block writes it so it can stand on its own. 143,966 characters is
// deliberate: the default chunk size is 65,536 bytes, so it arrives in
// exactly three pieces — two full ones and a remainder.
fs.writeFileSync('big.txt', 'x'.repeat(143966));

// createReadStream gives you the file PIECE BY PIECE instead of all at once.
// Passing 'utf8' makes each chunk a string instead of a raw Buffer.
const stream = fs.createReadStream('big.txt', 'utf8');

let totalChars = 0;

// 'data' fires once for every chunk Node reads off the disk.
stream.on('data', (chunk) => {
  totalChars += chunk.length;
  console.log('Got a chunk of', chunk.length, 'characters');
});

// 'end' fires once, after the LAST chunk has arrived.
stream.on('end', () => {
  console.log('Done. Total characters:', totalChars);
});

// Always handle 'error' — a missing file would otherwise crash the process.
stream.on('error', (err) => {
  console.error('Stream failed:', err.message);
});

Your turn. The program below works once you fill in the three blanks marked ___. Follow the 👉 hints, then run it and compare with the expected output.

// 🎯 YOUR TURN — replace each ___ then run it with: node turn.js

// 1) Make a Buffer from a string of your choice
const buf = Buffer.from(___);   // 👉 put text in 'single quotes', e.g. 'stream'

// 2) Log how many BYTES it holds
console.log(buf.___);           // 👉 the property that gives the byte count

// 3) Log it back as readable text
console.log(buf.___());         // 👉 the method that decodes bytes to a string

// ✅ Expected output (for 'stream'):
//    6
//    stream

3️⃣ Writable Streams & .pipe()

A writable stream is the opposite of a readable one: it's a destination you push chunks into. You create one with fs.createWriteStream(path). The magic happens when you connect a readable stream to a writable one with .pipe():

• • source.pipe(dest) — forwards every chunk from source straight into dest, then closes the destination when the source ends.
• • It also handles backpressure for you: if the destination (a slow disk, say) can't keep up, pipe automatically pauses the source until it catches up.
• • The writable side emits 'finish' when every byte has been flushed.

This is how you copy a multi-gigabyte file using only a tiny, constant amount of memory — the whole copy is just source.pipe(dest).

const fs = require('fs');

// Same stand-in file as the read example above, so this block runs alone too.
fs.writeFileSync('big.txt', 'x'.repeat(143966));

// A Readable stream: where the data comes FROM.
const source = fs.createReadStream('big.txt');

// A Writable stream: where the data goes TO.
const dest = fs.createWriteStream('copy.txt');

// pipe() connects them: every chunk read from source is written to dest,
// and Node automatically handles backpressure (pausing when dest is busy).
source.pipe(dest);

// 'finish' fires on the WRITABLE side when all data has been flushed.
dest.on('finish', () => {
  console.log('File copied with constant, low memory use!');
});

source.on('error', (err) => console.error('Read error:', err.message));
dest.on('error', (err) => console.error('Write error:', err.message));

Mini-Challenge: Count a File's Bytes

No blanks this time — just a brief and an outline to keep you on track. Stream a file, total up the byte length of each chunk, and print the result. Write it yourself, run it, and check your output against the example in the comments.

const fs = require('fs');

// 🎯 MINI-CHALLENGE: stream a file and count its bytes
// 1. Open 'notes.txt' with fs.createReadStream (no encoding = raw Buffers).
// 2. On each 'data' chunk, add chunk.length to a running total.
// 3. On 'end', print the total number of bytes.
// 4. On 'error', print the error message so a missing file won't crash you.
//
// ✅ Example output:
//    notes.txt is 2048 bytes

// your code here

Common Errors (and the fix)

• Reading a huge file with fs.readFileSync — this loads the entire file into memory at once, so a large file blows up your RAM (and Sync also freezes your whole program). Fix: use fs.createReadStream and process chunks as they arrive.
• Forgetting the 'end' event — if your final result depends on having seen the whole file, computing it inside 'data' gives wrong answers. Fix: accumulate in 'data' and report the result in 'end', which fires once at the very end.
• Not handling the 'error' event — stream errors aren't thrown, they're emitted. An unhandled 'error' crashes the whole Node process. Fix: add stream.on('error', fn) to every stream.
• Buffer encoding mismatches — calling buf.toString('ascii') on UTF-8 bytes (or counting .length as characters) garbles non-English text and emoji. Fix: stick to 'utf8' for text and remember .length is bytes, not characters.
• Ignoring backpressure — writing to a destination faster than it can accept (looping dest.write() without checking its return value) balloons memory. Fix: use .pipe(), which manages backpressure for you automatically.

Pro Tips

• 💡 Prefer pipeline() over .pipe() in real apps: require('stream').pipeline wires streams together and forwards errors and cleanup for you, avoiding leaked file handles.
• 💡 Set the encoding once: passing 'utf8' to createReadStream means your 'data' chunks are strings, so you don't have to call .toString() on every one.
• 💡 Buffers are bytes, not characters: when measuring text size for the network, Buffer.byteLength(str) gives the true byte count that str.length won't.

📋 Quick Reference — Streams & Buffers

🎉 Lesson Complete!

• ✅ A Buffer holds a fixed-length chunk of raw bytes; Buffer.from and .toString() convert to and from text
• ✅ A stream delivers data piece by piece instead of all at once
• ✅ fs.createReadStream + the 'data' and 'end' events read large files with low memory
• ✅ source.pipe(dest) copies streams efficiently and handles backpressure
• ✅ Always attach an 'error' handler — stream errors are emitted, not thrown
• ✅ Stream big files; only use readFile for small ones

Practice quiz

Continue this course

• Previous: Events & EventEmitter
• Next: The http Module — Make requests and understand the building blocks of HTTP in Node
• Quick reference: Node.js cheat sheet

Frequently asked questions

What is the difference between a stream and a Buffer in Node.js?

A Buffer is a single, fixed-length chunk of raw binary data sitting in memory right now. A stream is the moving pipeline that delivers those chunks one after another over time. Think of a Buffer as one bucket of water and a stream as the hose: the hose hands you bucket after bucket so you never have to hold the whole reservoir at once. Streams emit Buffers (or strings, if you set an encoding) on their 'data' event.

When should I use a stream instead of fs.readFile?

Use a stream whenever a file might be large or its size is unknown. fs.readFile (and fs.readFileSync) load the entire file into memory before you get a single byte, so a 2 GB file needs 2 GB of RAM and can crash your process. A read stream processes the file in small chunks (64 KB by default), so memory use stays low and constant no matter how big the file is. For small config files, readFile is simpler and perfectly fine.

What does stream.pipe() actually do?

pipe() wires a readable stream's output straight into a writable stream's input. It forwards every chunk for you, ends the destination when the source finishes, and — crucially — handles backpressure automatically by pausing the readable side when the writable side can't keep up. It is the simplest and safest way to move data from one stream to another, which is why source.pipe(dest) is the idiomatic way to copy or transform files.

Why do I need to listen for the 'error' event on a stream?

Stream errors are not thrown like normal exceptions — they are emitted as an 'error' event. If you never attach a handler with stream.on('error', ...), an emitted error becomes an uncaught exception that crashes the whole Node process. Always add an error listener to every readable and writable stream so problems like a missing file or a full disk are handled gracefully instead of taking your app down.