Serving ML Models

Turn a trained model into a reliable production service — REST and gRPC inference endpoints, request batching, autoscaling, model versioning, and safe rollouts with canary and A/B deploys.

Learn Serving ML Models in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🍽️ Real-World Analogy: A Restaurant Kitchen

A trained model in a notebook is like a chef who only cooks at home. Serving it is opening a restaurant: the same cooking skill, but now you must take orders, cook many at once, and never keep a table waiting.

The whole lesson is about running that kitchen well: fast tickets, full grills, enough cooks, and a safe way to change the menu.

1 Inference Endpoints — REST and gRPC

Serving means putting your model behind a network address so other programs send an input and get a prediction back. The address is an inference endpoint .

A common setup is REST on the edge for callers, gRPC between internal services. Whichever you pick, two endpoints are non-negotiable: a /predict for inference and a /health the load balancer can poll.

Worked example — a minimal REST endpoint with FastAPI:

Notice the model loads once at startup and is reused by every request. The /health route is how the kitchen tells the maitre d' (load balancer) it is ready for orders.

2 Serving Frameworks — Don't Build the Plumbing Yourself

FastAPI is great for one model, but a dedicated serving framework gives you batching, model versioning, multi-model hosting, and GPU scheduling for free. Pick by your stack:

Framework

Built for

Reach for it when…

3 Request Batching, Latency vs Throughput

A GPU runs a batch of 8 inputs almost as fast as a single one. Request batching groups incoming requests into one forward pass, so throughput (requests per second) shoots up. The cost: a request waits a few milliseconds for the batch to fill, so its latency rises slightly. Frameworks expose two dials — max batch size and max wait — to balance the two.

You measure latency in percentiles , not averages. p50 is the median request; p99 is the slow tail — 1 in 100 requests is at least this slow. Users feel the tail, so p99 is the number that matters for an SLA.

Worked example — the batching loop a framework runs for you:

The batch flushes when it is full or the wait timer fires — that timer is what keeps p99 from exploding under light traffic.

▶️ Worked Example: Latency Percentiles (run it)

Run this to see how one slow request pulls p99 far above p50, even when most requests are fast. Read the comments, then press run.

🎯 Your Turn #1: Batch the Requests

Fill in the two blanks marked ___ . Group 7 requests into batches of 4 and count how many GPU passes that takes. Check your output against the # ✅ Expected output comment.

🎯 Your Turn #2: Measure the Tail

Fill in the two blanks so percentile() sorts the timings and returns the right value. A single 150ms request should send p99 sky-high while p50 stays calm.

4 Autoscaling and Cold Starts

Autoscaling adds or removes server replicas as traffic changes — more cooks at the dinner rush, fewer at 3am. You scale on a signal like GPU utilisation, queue depth, or requests per second.

The catch is the cold start : a freshly added replica is slow on its first request because the model still has to load into memory and the GPU warm up. Two fixes: keep a minimum number of replicas always running , and run a warmup inference before the replica accepts traffic (you saw the warmup call in Section 1).

5 Model Versioning, Canary & A/B Deploys

Never overwrite a live model. Give every model a version (v1, v2, …) so you can serve a specific one, compare them, and roll back instantly if v2 misbehaves.

Trial the new recipe on a few tables (canary), or serve two recipes to see which sells better (A/B) — either way, the old recipe is one switch away.

Common Errors (And How to Fix Them)

These five mistakes sink most first serving deployments:

One request per forward pass leaves the GPU 90% idle and caps your throughput.

✅ Fix: enable dynamic batching (set max batch size + max wait), or use a framework that does it for you.

Loading the model inside the handler reloads 500MB+ from disk per call — 10s latencies.

✅ Fix: load once at startup, run a warmup inference, and keep a minimum replica count.

Overwriting the live model means a bad deploy has no undo and no way to compare.

✅ Fix: tag every model with a version and deploy via canary so rollback is one switch.

A synchronous DB or network call inside the handler stalls the whole worker under load.

✅ Fix: use async handlers, move slow work off the hot path, and set request timeouts.

An infinite request queue hides overload: latency climbs forever and memory blows up instead of failing fast.

✅ Fix: cap the queue length and reject extra requests with HTTP 429 so callers can back off.

📋 Quick Reference

Concept

What it is

Why it matters

❓ Frequently Asked Questions

🎯 Mini-Challenge: SLA Checker

Now write it yourself with only a comment outline. Build a tiny SLA checker that flags when your p99 latency breaches a 200ms budget. The starter has just the steps — no filled-in logic.

Lesson 42 complete — you can serve a model in production!

You can expose a model over REST or gRPC, pick a serving framework, batch requests for throughput, read p50/p99 latency, autoscale without cold starts, and roll out new versions safely with canary and A/B deploys.

🚀 Up next: Model Monitoring — watch your live model for drift, bias, and degradation before your users notice.

Practice quiz

What is model serving?

  • Training a model on more data
  • Compressing a model to fewer bits
  • Wrapping a trained model behind a network endpoint so other programs can get predictions
  • Cleaning the training data

Answer: Wrapping a trained model behind a network endpoint so other programs can get predictions. Serving exposes a trained model over a network (REST or gRPC) so clients can send inputs and receive predictions.

When is REST/JSON usually preferred over gRPC for inference?

  • When you want something easy to debug and call from any client, including browsers
  • When you need the lowest possible latency for tensors
  • When callers are only other internal services
  • When payloads are large binary blobs

Answer: When you want something easy to debug and call from any client, including browsers. REST/JSON is human-readable and easy to debug or call from a browser; gRPC suits fast internal service-to-service calls.

Why does request batching improve serving throughput?

  • It reduces model accuracy
  • It loads the model faster
  • It removes the need for a health check
  • A GPU runs a batch almost as fast as a single item, so grouping requests raises requests per second

Answer: A GPU runs a batch almost as fast as a single item, so grouping requests raises requests per second. Grouping many inputs into one forward pass uses the GPU efficiently, dramatically increasing throughput.

What is the main trade-off introduced by request batching?

  • Lower accuracy
  • A request may wait a few milliseconds for the batch to fill, raising its latency slightly
  • Higher memory usage forever
  • It disables versioning

Answer: A request may wait a few milliseconds for the batch to fill, raising its latency slightly. Batching raises throughput but a request waits for the batch to fill, slightly increasing per-request latency.

Why report latency as p50 and p99 percentiles instead of an average?

  • The slow tail (p99) is what users feel, and an average hides it
  • Averages are impossible to compute
  • Percentiles are always lower
  • p99 ignores slow requests

Answer: The slow tail (p99) is what users feel, and an average hides it. A few slow requests (the tail) matter most to users; p99 captures that tail while an average can hide it.

What is a cold start in model serving?

  • A request that returns an error
  • A request sent to the wrong version
  • The slow first request after a server boots, because the model must load and the GPU warm up
  • A batch that never fills

Answer: The slow first request after a server boots, because the model must load and the GPU warm up. A cold start is the slow first request after boot or scale-up while the model loads into memory.

Which practices help avoid cold starts?

  • Load the model inside every request handler
  • Load the model once at startup, run a warmup inference, and keep a minimum number of replicas
  • Scale to zero replicas always
  • Disable health checks

Answer: Load the model once at startup, run a warmup inference, and keep a minimum number of replicas. Loading once, warming up, and keeping minimum replicas running prevents the slow first-request penalty.

What does autoscaling do?

  • Versions the model automatically
  • Compresses the model
  • Batches requests
  • Adds or removes server replicas as traffic changes

Answer: Adds or removes server replicas as traffic changes. Autoscaling adjusts the number of replicas based on a signal like GPU utilisation or requests per second.

What is the difference between a canary deploy and an A/B deploy?

  • Canary measures which model is better; A/B is a safety check
  • Canary sends a small slice of traffic to a new version as a safety check; A/B splits traffic to measure which performs better
  • They are identical
  • Canary requires no rollback plan

Answer: Canary sends a small slice of traffic to a new version as a safety check; A/B splits traffic to measure which performs better. A canary is a safety mechanism (small slice, check health); an A/B deploy is a measurement mechanism comparing versions.

Why give every model a version when serving?

  • To make the model smaller
  • To increase latency
  • So you can serve a specific version, compare versions, and roll back instantly if a new one misbehaves
  • Because REST requires it

Answer: So you can serve a specific version, compare versions, and roll back instantly if a new one misbehaves. Versioning lets you compare models and instantly roll back a bad deploy instead of overwriting the live model.

Continue this course