Lesson 43 • Advanced
Monitoring Models in Production 📊
Deploying a model is just the beginning. Learn to detect data drift, outliers, bias, and performance degradation before they cause real damage.
What You'll Learn in This Lesson
- • How to detect data drift using PSI (Population Stability Index)
- • Monitoring for concept drift with accuracy tracking over time
- • Catching anomalous inputs before they corrupt predictions
- • Fairness monitoring with disparate impact and equal opportunity metrics
- • Building automated alerting pipelines for production ML
🎯 Real-World Analogy: Think of model monitoring like a car dashboard. Your engine (model) might run fine today, but without gauges for temperature, oil pressure, and fuel — you won't know something's wrong until the engine fails. Monitoring gives you those gauges for your ML system.
1️⃣ Why Models Degrade
A model that was 95% accurate at deployment can silently drop to 70% within weeks. The three main causes:
| Type | What Changes | Example |
|---|---|---|
| Data Drift | Input distribution shifts | New customer demographics |
| Concept Drift | Input→output relationship changes | "Good credit" threshold shifts |
| Feature Drift | Feature pipeline breaks | API returns null for a column |
💡 Pro Tip: Monitor all three — most teams only check accuracy and miss silent data issues.
Try It: Data Drift Detection
Calculate PSI to detect when your input data shifts from the training distribution
Python
import numpy as np
# ============================================
# DATA DRIFT DETECTION
# ============================================
# Data drift = input distribution changes over time
# Model was trained on summer data, now it's winter
np.random.seed(42)
print("=== Population Stability Index (PSI) ===")
print()
print("PSI measures how much the input distribution shifted.")
print("Think of it like checking if your customers changed.")
print()
def calculate_psi(reference, current, bins=10)
...2️⃣ Monitoring Architecture
A production monitoring system has these layers:
┌──────────────────────────────────────┐ │ Alerting (PagerDuty/Slack) │ ├──────────────────────────────────────┤ │ Dashboard (Grafana/DataDog) │ ├──────────────────────────────────────┤ │ Metric Store (Prometheus/BigQuery) │ ├──────────────────────────────────────┤ │ Collectors (input/output loggers) │ ├──────────────────────────────────────┤ │ Model Inference Service │ └──────────────────────────────────────┘
Key metrics to track:
- Latency: p50, p95, p99 response times
- Throughput: Requests per second
- Error rate: Failed predictions / total
- Feature distributions: Mean, std, min, max per feature
- Prediction distribution: Class balance or output range
Try It: Anomaly Detection in Production
Catch outlier inputs using z-score monitoring before they corrupt predictions
Python
import numpy as np
# ============================================
# OUTLIER & ANOMALY DETECTION IN PRODUCTION
# ============================================
np.random.seed(42)
print("=== Production Anomaly Detection ===")
print()
print("Your model expects certain input ranges.")
print("Outliers can cause silent failures or wrong predictions.")
print()
# Simulate normal feature distributions from training
feature_names = ["age", "income", "credit_score", "loan_amount"]
train_means = [35, 55000
...Try It: Bias Monitoring
Track fairness metrics across demographic groups with disparate impact analysis
Python
import numpy as np
# ============================================
# BIAS MONITORING IN PRODUCTION
# ============================================
np.random.seed(42)
print("=== Fairness Metrics Monitoring ===")
print()
print("Even a fair model at training can become biased in production")
print("as user demographics shift over time.")
print()
# Simulate loan approval predictions across groups
groups = {
"Group A (majority)": {"total": 5000, "approved": 3800, "default_rate": 0.05},
"Grou
...3️⃣ Common Mistakes
⚠️
Only monitoring accuracy — data drift happens before accuracy drops. By the time accuracy falls, damage is done.
⚠️
No baseline comparison — always compare current metrics against a reference window, not just absolute thresholds.
⚠️
Alert fatigue — too many alerts and teams ignore them. Tier alerts: info → warning → critical.
💡
Pro Tip: Store raw predictions and inputs for at least 30 days. When something breaks, you'll need them to debug.
📋 Quick Reference — Model Monitoring
| Metric | Tool | Alert Threshold |
|---|---|---|
| PSI | Evidently, WhyLabs | > 0.25 → retrain |
| Accuracy | MLflow, Prometheus | < baseline - 5% |
| Latency p99 | Grafana, DataDog | > 200ms |
| Disparate Impact | Fairlearn, AIF360 | < 0.8 ratio |
| Feature Null % | Great Expectations | > 5% nulls |
🎉 Lesson Complete!
You can now monitor ML models in production! Next, learn how to automate the entire ML lifecycle with MLOps pipelines.
Sign up for free to track which lessons you've completed and get learning reminders.