Courses/C#/Streams, Buffers & Pipelines

Advanced Track

Streams, Buffers & Pipelines

By the end of this lesson you'll be able to read and write data with C#'s stream types — StreamReader, StreamWriter, BufferedStream and MemoryStream — dispose them cleanly with using, and understand why flushing and rewinding matter. These are the tools behind every file, network and pipeline read in .NET.

What You'll Learn

Explain what a Stream is — a one-way flow of bytes from a source
Write and read text with StreamWriter and StreamReader
Use File.ReadAllText / WriteAllText for quick whole-file work
Wrap a stream in a BufferedStream to batch small operations
Use MemoryStream to read and write entirely in memory
Dispose streams safely with using, and know when to Flush and rewind

Before you start: make sure you've finished Logging Architecture. You should already be comfortable with classes, generics, and the using statement, because every stream type here implements IDisposable and must be cleaned up.

💡 Real-World Analogy

A stream is a conveyor belt in a factory. Data flows past you piece by piece — you never need to see the whole shipment at once, which is how a program can process a file far bigger than its memory. A buffer is the loading dock beside the belt: items pile up there and get moved in one bulk trip instead of one box at a time, which is far more efficient. A StreamWriter is the worker placing boxes on the belt; a StreamReader is the worker taking them off. And Flush is shouting "send what's on the dock right now!" — until you do, the last few boxes may still be sitting there, not yet on the belt.

The Stream Family at a Glance

Every I/O type in .NET is built on the abstract Stream class — a flow of bytes you can read, write, and (sometimes) seek within. Different streams have different sources, and helper classes layer text or buffering on top. Here's the cast you'll meet in this lesson.

Type	What it does	Reach for it when
FileStream	Raw bytes to/from a file on disk	You need a real file handle
MemoryStream	A stream backed by a byte array in RAM	Tests, buffers, no disk needed
StreamWriter	Turns your text into bytes for a stream	Writing lines of text
StreamReader	Turns a stream's bytes back into text	Reading lines of text
BufferedStream	Batches small reads/writes into big ones	Many tiny operations are slow
File.ReadAllText	Whole file in / out as one string	Small files, quick scripts

The clever part: because they all share the Stream base type, a StreamReader doesn't care whether it's reading a file, a network socket, or memory. Learn the pattern once and it works everywhere.

1. What a Stream Is — Write, Flush, Rewind, Read

A stream is a one-way flow of bytes with an internal cursor marking your current position. You write bytes in or read bytes out, and the cursor moves along as you go. A StreamWriter sits on top and converts your text into bytes; a StreamReader converts bytes back into text. To keep this lesson runnable without a real file, we use a MemoryStream — a stream whose source is just an array in memory. Read this worked example carefully: it shows the full round trip and the two steps beginners forget — Flush and rewinding the cursor.

Worked example: write to memory, then read it back

Read every comment and run it — note the Flush and the rewind to Position 0.

Try it Yourself »

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // A STREAM is a one-way flow of bytes you read from or write to.
        // The SOURCE can be a file, the network, or — here — memory.
        // We use a MemoryStream so this runs anywhere, with no real file.

        using var ms = new MemoryStream();   // an empty, growable byte buffer

        // === WRITE ===
        // A StreamWriter turns your text into bytes and pushes them in.
      
...

Your turn. The program below does the same round trip but is missing two pieces. Fill in the two ___ blanks using the hints, then run it. If you see nothing printed, you forgot to rewind.

🎯 Your turn: a memory round trip

Flush the writer and rewind to position 0, then read your line back.

Try it Yourself »

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — fill in the blanks marked with ___, then run it.
        // Goal: write one line into memory, rewind, and read it back.

        using var ms = new MemoryStream();

        // 1) Write a line of text into the stream.
        using (var writer = new StreamWriter(ms, Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("Saved without touching the disk!");
   
...

2. Reading & Writing Files — Helpers and StreamReader

For small files, .NET gives you one-line helpers: File.WriteAllText replaces a file's contents, and File.ReadAllText / File.ReadAllLines pull it all back. They're perfect for config files and quick scripts. But they load the entire file into memory at once, so for large files you read line by line with a StreamReader inside a using block. The using is not optional politeness — it guarantees the operating-system file handle is closed even if an exception is thrown halfway through.

Worked example: File helpers + StreamReader with using

Write a file, read it whole, then read it line by line — and see how using cleans up.

Try it Yourself »

using System;
using System.IO;

class Program
{
    static void Main()
    {
        string path = "notes.txt";

        // === Whole-file helpers — shortest path for SMALL files ===
        // WriteAllText replaces the file's entire contents in one call.
        File.WriteAllText(path, "First line\nSecond line\nThird line");

        // ReadAllText pulls the WHOLE file into one string.
        string everything = File.ReadAllText(path);
        Console.WriteLine($"Characters in file: {everythin
...

Now you try. The buffer below already has three lines written into it; your job is to read them back and process them by counting how many there are. Fill in the two ___ blanks:

🎯 Your turn: read multiple lines and count them

Loop with ReadLine() until null, count the lines, and print the total.

Try it Yourself »

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — count how many lines are in an in-memory buffer.
        // The data is already written for you; you just read and count.

        using var ms = new MemoryStream();
        using (var writer = new StreamWriter(ms, Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("apple");
            writer.WriteLine("banana");
            writer.WriteLine("cherry");
 
...

3. Buffering — Doing Fewer, Bigger Trips

Every individual read or write to a raw stream can cost a trip to the disk or network — and those trips have fixed overhead, so a thousand tiny writes are far slower than a few big ones. A BufferedStream wraps another stream and accumulates small operations in memory, flushing them out in large chunks. You don't change what gets written — only how efficiently. (Note: StreamWriter and FileStream already buffer internally, so you mainly add a BufferedStream around streams that don't, such as a raw network stream.)

Worked example: a BufferedStream around a MemoryStream

Buffer three writes, flush, and read them back unchanged.

Try it Yourself »

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // === BufferedStream — fewer, bigger trips to the source ===
        // Every read/write to a raw stream can be an expensive system call.
        // BufferedStream wraps another stream and batches small operations
        // into large ones, so 1,000 tiny writes become a handful of big ones.
        using var ms = new MemoryStream();
        using (var buffered = new BufferedStream(ms, buffer
...

🔎 Deep Dive: why `using`, Flush, and rewind matter

Streams hold unmanaged resources — OS file handles, network sockets, locked buffers. The garbage collector cleans up normal C# objects, but it can't promptly release these. That's what IDisposable and using are for: the moment a using block ends, Dispose() runs, which flushes any buffered data and frees the handle — even if an exception was thrown.

Disposing a StreamWriter normally flushes and then closes the underlying stream too. That's why the examples pass leaveOpen: true — so disposing the writer flushes the text but leaves the MemoryStream open for us to rewind and read.

using var ms = new MemoryStream();          // disposed at end of method
using (var w = new StreamWriter(ms, leaveOpen: true))
{
    w.WriteLine("data");
    w.Flush();        // push text out NOW; Dispose would also do this
}                     // 👈 writer disposed here — ms stays open
ms.Position = 0;      // rewind — without this, the read returns nothing
// ... read ms ...

Two failure modes to remember: forget to Flush/dispose and your last writes never reach the stream; forget to set Position = 0 and the reader starts at the end, reading nothing.

Putting It Together: a Mini CSV Reader

Here's a small but real program that uses everything from this lesson at once — a MemoryStream built straight from text, a StreamReader in a using, line-by-line reading, and parsing each line into a total. This is exactly how you'd process a real CSV file; only the source would change from memory to a FileStream.

Worked example: total an order from a CSV buffer

Parse product,quantity,price lines and sum the order total.

Try it Yourself »

using System;
using System.Globalization;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // === Mini "CSV" reader — total an order from an in-memory buffer ===
        // Each line is  product,quantity,unitPrice
        string data = "Keyboard,2,29.99\nMouse,1,14.50\nMonitor,1,189.00";

        // Build a stream straight from the text's UTF-8 bytes.
        using var ms = new MemoryStream(Encoding.UTF8.GetBytes(data));
        using var reader = new Str
...

Swapping new MemoryStream(...) for File.OpenRead("orders.csv") turns this into a real file reader — the StreamReader loop doesn't change a single line. That's the power of programming against the Stream abstraction.

Beyond Streams: System.IO.Pipelines

When you need the absolute highest throughput — a web server parsing millions of request bodies, say — even buffered streams leave performance on the table. System.IO.Pipelines is a newer API built for exactly that. A Pipe has a PipeWriter and a PipeReader that share a pool of memory buffers, giving you zero-copy reads (no shuffling bytes between arrays), automatic back-pressure (the writer slows down if the reader falls behind), and far less garbage-collector pressure.

It's the engine inside Kestrel, ASP.NET Core's web server. You won't reach for it in everyday code — a StreamReader is simpler and plenty fast — but it's the tool to know about the day a profiler says stream parsing is your bottleneck.

// Sketch only — Pipelines trades simplicity for raw speed:
var pipe = new Pipe();
PipeWriter writer = pipe.Writer;   // fills shared buffers
PipeReader reader = pipe.Reader;   // reads them with zero copying
// reader.ReadAsync() hands you a ReadOnlySequence<byte> to parse in place

Pro Tips

💡 Always wrap streams in using: it disposes them — flushing and closing the handle — even when an exception is thrown.
💡 Rewind before you read: after writing to a MemoryStream, set Position = 0 or the reader starts at the end and gets nothing.
💡 Use leaveOpen: true when you want to keep a stream alive after disposing a StreamWriter/StreamReader wrapped around it.
💡 Read big files line by line with StreamReader.ReadLine() instead of File.ReadAllText to avoid loading gigabytes into memory.
💡 Prefer the async versions (ReadLineAsync, WriteLineAsync) for files in server apps so you don't block a thread on slow I/O.

Common Errors (and the fix)

"System.IO.FileNotFoundException: Could not find file '...'": you tried to open a file for reading that doesn't exist (wrong path or wrong working directory). Check the path, and guard reads with if (File.Exists(path)) before opening.
Nothing comes out / empty read: you wrote to a MemoryStream but forgot ms.Position = 0; before reading. The cursor is at the end, so there's nothing left to read. Rewind first.
Last lines missing from the file: you didn't dispose or Flush the StreamWriter, so its buffer never reached the stream. Wrap it in using (or call Flush()) so the buffered text is written.
"System.ObjectDisposedException: Cannot access a closed Stream": you used a stream after its using block ended, or a StreamWriter closed the underlying stream. Keep usage inside the block, or pass leaveOpen: true.
"System.IO.IOException: The process cannot access the file ... because it is being used by another process": a previous handle wasn't released. This is exactly what using prevents — make sure every stream is disposed.

📋 Quick Reference

Task	Code	Notes
Write a whole file	File.WriteAllText(path, text)	Replaces contents
Read a whole file	File.ReadAllText(path)	One string; small files
Read all lines	File.ReadAllLines(path)	Returns a string[]
Read line by line	while ((l = r.ReadLine()) != null)	Scales to huge files
In-memory stream	new MemoryStream()	No disk; great for tests
Rewind a stream	ms.Position = 0;	Before reading what you wrote
Push buffered data	writer.Flush();	Dispose does this too
Auto-dispose	using var r = new StreamReader(s);	Closes on scope exit

Frequently Asked Questions

Q: When should I use File.ReadAllText instead of a StreamReader?

Use File.ReadAllText for small files where loading everything at once is fine — config, small data files, quick scripts. Switch to a StreamReader with ReadLine() when the file is large, so you process one line at a time instead of pulling gigabytes into memory.

Q: Why does my MemoryStream read return nothing?

Almost always because you didn't rewind. Writing leaves the cursor at the end of the stream; reading from there yields nothing. Set ms.Position = 0; after writing and before reading.

Q: Do I really need Flush() if I'm using using?

Disposing (which using does at the end of the block) flushes for you, so an explicit Flush() is only needed when you want the data pushed out before the block ends — for example, to rewind and read the same stream straight away.

Q: What's the difference between a Stream and a StreamReader?

A Stream deals in raw bytes. A StreamReader/StreamWriter wraps a stream and handles the text encoding for you, so you work with strings and lines instead of byte arrays.

Q: When would I use System.IO.Pipelines over streams?

Only in high-throughput, performance-critical code — parsing huge volumes of network data, for instance. For everyday file work a StreamReader is simpler and fast enough; Pipelines trades that simplicity for zero-copy reads and back-pressure.

Mini-Challenge: Average a Column

No blanks this time — just a brief and an outline to keep you on track. Read the in-memory buffer of numbers line by line, total them, count them, and print the average. Run it and check your output against the expected line in the comments. This is the same loop you'd use over a real file of readings or scores.

🎯 Mini-Challenge: compute the average from a buffer

Read each number with ReadLine(), sum and count them, then print the average.

Try it Yourself »

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 MINI-CHALLENGE: average a column of numbers from a buffer
        // The data below is one number per line. Compute the AVERAGE.
        //
        // 1. Make a MemoryStream from these UTF-8 bytes (done for you).
        // 2. Wrap it in a StreamReader.
        // 3. Loop with ReadLine() until it returns null.
        //    - Parse each line with int.Parse(line)
        //    - Add it to
...

🎉 Lesson Complete

✅ A stream is a one-way flow of bytes with a moving cursor
✅ StreamWriter writes text as bytes; StreamReader reads bytes back as text
✅ File.ReadAllText / WriteAllText are the quick path for small files
✅ BufferedStream batches small operations into big ones for speed
✅ MemoryStream gives you a stream with no disk — ideal for tests and buffers
✅ using disposes streams safely; Flush and Position = 0 are the round-trip essentials
✅ System.IO.Pipelines is the zero-copy, back-pressure-aware option for extreme throughput
✅ Next lesson: JSON Processing — serialise and deserialise data with System.Text.Json

Streams, Buffers & Pipelines

What You'll Learn

💡 Real-World Analogy

The Stream Family at a Glance

1. What a Stream Is — Write, Flush, Rewind, Read

Worked example: write to memory, then read it back

🎯 Your turn: a memory round trip

2. Reading & Writing Files — Helpers and StreamReader

Worked example: File helpers + StreamReader with using

🎯 Your turn: read multiple lines and count them

3. Buffering — Doing Fewer, Bigger Trips

Worked example: a BufferedStream around a MemoryStream

🔎 Deep Dive: why using, Flush, and rewind matter

Putting It Together: a Mini CSV Reader

Worked example: total an order from a CSV buffer

Beyond Streams: System.IO.Pipelines

Pro Tips

Common Errors (and the fix)

📋 Quick Reference

Frequently Asked Questions

Mini-Challenge: Average a Column

🎯 Mini-Challenge: compute the average from a buffer

🎉 Lesson Complete

Cookie & Privacy Settings

🔎 Deep Dive: why `using`, Flush, and rewind matter