Serving ML Models

Serving means putting your model behind a network address so other programs send an input and get a prediction back. The address is an inference endpoint.

• REST/JSON — human-readable, works from a browser or curl, easy to debug. Best default for public APIs.
• gRPC — binary protocol, lower latency and smaller payloads for tensors. Best for fast service-to-service calls inside your cluster.

A common setup is REST on the edge for callers, gRPC between internal services. Whichever you pick, two endpoints are non-negotiable: a /predict for inference and a /health the load balancer can poll.

Worked example — a minimal REST endpoint with FastAPI:

# A minimal REST inference endpoint with FastAPI.
# Run with:  uvicorn app:app --host 0.0.0.0 --port 8000

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

# Load the model ONCE at startup, not inside the handler.
# 'model' lives for the life of the process and is reused by every request.
model = None

@app.on_event("startup")
def load_model():
    global model
    # In real code: model = torch.load("model.pt"); model.eval()
    model = lambda text: {"label": "positive", "score": 0.97}
    # Warmup: run one dummy inference so the first real request is fast.
    model("warmup")

# Pydantic validates the request body and rejects bad input early.
class PredictRequest(BaseModel):
    text: str

class PredictResponse(BaseModel):
    label: str
    confidence: float

@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
    out = model(req.text)                 # reuse the loaded model
    return PredictResponse(label=out["label"], confidence=out["score"])

@app.get("/health")                       # used by the load balancer
def health():
    return {"status": "ok", "model_loaded": model is not None}

# Expected output (POST /predict with {"text": "great movie"}):
#   {"label": "positive", "confidence": 0.97}
# Expected output (GET /health):
#   {"status": "ok", "model_loaded": true}

Notice the model loads once at startup and is reused by every request. The /health route is how the kitchen tells the maitre d' (load balancer) it is ready for orders.

FastAPI is great for one model, but a dedicated serving framework gives you batching, model versioning, multi-model hosting, and GPU scheduling for free. Pick by your stack:

A GPU runs a batch of 8 inputs almost as fast as a single one. Request batching groups incoming requests into one forward pass, so throughput (requests per second) shoots up. The cost: a request waits a few milliseconds for the batch to fill, so its latency rises slightly. Frameworks expose two dials — max batch size and max wait — to balance the two.

You measure latency in percentiles, not averages. p50 is the median request; p99 is the slow tail — 1 in 100 requests is at least this slow. Users feel the tail, so p99 is the number that matters for an SLA.

Worked example — the batching loop a framework runs for you:

# Why serving frameworks batch requests: one GPU pass handles many inputs.
# This sketches the loop TorchServe / Triton / vLLM run for you.

import time

MAX_BATCH = 8        # never let a batch grow unbounded
MAX_WAIT_MS = 5      # don't make request #1 wait too long for a full batch

def run_batch(texts):
    # One forward pass over the whole batch — ~same cost as a single item.
    return ["positive"] * len(texts)

def serve(queue):
    results = []
    start = time.time()
    batch = []
    for item in queue:                         # incoming requests
        batch.append(item)
        waited_ms = (time.time() - start) * 1000
        # Flush when the batch is full OR we've waited long enough.
        if len(batch) >= MAX_BATCH or waited_ms >= MAX_WAIT_MS:
            results += run_batch(batch)         # 1 GPU pass for the whole batch
            batch = []
            start = time.time()
    if batch:
        results += run_batch(batch)             # flush the leftovers
    return results

print(serve(["a", "b", "c"]))

# Expected output:
#   ['positive', 'positive', 'positive']

The batch flushes when it is full or the wait timer fires — that timer is what keeps p99 from exploding under light traffic.

Autoscaling adds or removes server replicas as traffic changes — more cooks at the dinner rush, fewer at 3am. You scale on a signal like GPU utilisation, queue depth, or requests per second.

The catch is the cold start: a freshly added replica is slow on its first request because the model still has to load into memory and the GPU warm up. Two fixes: keep a minimum number of replicas always running, and run a warmup inference before the replica accepts traffic (you saw the warmup call in Section 1).

Never overwrite a live model. Give every model a version (v1, v2, …) so you can serve a specific one, compare them, and roll back instantly if v2 misbehaves.

• Canary deploy — route a small slice (e.g. 5%) of traffic to v2. If error rate and latency stay healthy, ramp to 100%; if not, roll back. This is a safety mechanism.
• A/B deploy — split traffic between v1 and v2 on purpose and measure which scores better on a business metric. This is a measurement mechanism.
• Shadow mode — send v2 a copy of real traffic but don't return its answers, so you can compare offline with zero user risk.

Trial the new recipe on a few tables (canary), or serve two recipes to see which sells better (A/B) — either way, the old recipe is one switch away.

These five mistakes sink most first serving deployments: