Model Interpretability (SHAP & LIME)

Reviewed & published by Brayan K

Open the black box. Learn to explain why a model made a decision using feature importance, SHAP values, LIME, and partial dependence — the difference between a model you trust and one you don't.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• White-box vs black-box models
• Global feature importance
• How SHAP values explain a single prediction
• How LIME builds a local explanation
• Reading a partial dependence plot
• Why interpretability matters in regulated fields

🎯 Real-World Analogy: The Loan Officer's Explanation

Imagine a bank rejects your loan. "Computer says no" isn't good enough — by law and by fairness you deserve to know why. A good loan officer explains: "Your income helped, but three missed payments last year pushed you over the line." That breakdown is exactly what SHAP and LIME produce for a model.

Interpretability turns an opaque score into a sentence a human can check, challenge, and trust — and that's what separates a deployable model from a risky one.

1 Black-Box vs Interpretable Models

Some models are white-box — you can read the decision straight out of them. Linear and logistic regression expose coefficients; a small decision tree is a flowchart. Others are black-box — deep neural networks and large gradient-boosting ensembles are often the most accurate, but their internal reasoning is opaque.

The tension is real: the most accurate model is frequently the least interpretable. Rather than always downgrading to a simpler model, we use post-hoc, model-agnostic explanation tools to interrogate the black box after training.

Two kinds of explanation:

• Global: which features matter across the whole model (feature importance, PDP).
• Local: why this one prediction came out the way it did (SHAP, LIME).

2 Feature Importance & SHAP Values

Global feature importance ranks which features matter most overall. But it can't tell you why a specific customer was rejected. SHAP values can: borrowed from cooperative game theory (Shapley values), they fairly split a single prediction among its features.

Their defining property is additivity: for one prediction, the SHAP values sum to the gap between that prediction and the model's baseline (average) output. So you can read exactly which features pushed the result up and which pulled it down. Run this hand-built version.

# A tiny additive explanation, SHAP-style (plain Python, no libraries).
# Idea: a prediction = baseline (average) + each feature's contribution.
# The contributions ADD UP to (prediction - baseline).

baseline = 0.30          # the model's average predicted risk

# How much each feature pushed THIS prediction up (+) or down (-).
contributions = {
    "income":   -0.10,   # higher income lowered the risk
    "age":       0.05,   # age nudged it up a little
    "defaults":  0.25,   # past defaults pushed risk up a lot
}

prediction = baseline + sum(contributions.values())
print("baseline:", baseline)
print("prediction:", round(prediction, 2))

# The contributions explain the gap from baseline to prediction.
gap = round(prediction - baseline, 2)
print("sum of contributions:", round(sum(contributions.values()), 2), "== gap", gap)

# Rank features by absolute impact (the biggest driver first).
ranked = sorted(contributions.items(), key=lambda kv: abs(kv[1]), reverse=True)
print("top driver:", ranked[0][0])

# Expected output:
# baseline: 0.3
# prediction: 0.5
# sum of contributions: 0.2 == gap 0.2
# top driver: defaults

3 LIME & Partial Dependence

LIME (Local Interpretable Model-agnostic Explanations) takes a different route: to explain one prediction, it nudges the input around that point, watches how the black box reacts, and fits a simple model locally to approximate it. The simple model's coefficients become the explanation — fast and intuitive, though only faithful near that one point.

A partial dependence plot (PDP) gives the global complement: it shows how the predicted outcome changes as one feature varies across its range, averaging over the others — revealing whether a feature's effect is steady, threshold-like, or non-linear.

🌍 Worked Example: The Real Tools (SHAP, LIME, PDP)

Here's how the libraries are called. Study it (it isn't runnable in the in-browser sandbox) and notice that all three are applied after training to an already-fitted model.

# WORKED EXAMPLE — what the real tools look like (SHAP & LIME).
# You don't run this here; study it so the library code feels familiar.

# ----- SHAP on a tree model -----
import shap

explainer = shap.TreeExplainer(model)        # fast for tree/boosting models
shap_values = explainer.shap_values(X)       # per-feature contributions
shap.summary_plot(shap_values, X)            # global view of feature impact
# For ONE row, shap_values[row] sums to (prediction - baseline).

# ----- LIME on any black-box model -----
from lime.lime_tabular import LimeTabularExplainer

lime_exp = LimeTabularExplainer(
    X_train, feature_names=cols, mode="classification"
)
explanation = lime_exp.explain_instance(X_test[0], model.predict_proba)
explanation.show_in_notebook()               # a simple LOCAL explanation

# ----- Partial dependence (scikit-learn) -----
from sklearn.inspection import PartialDependenceDisplay
PartialDependenceDisplay.from_estimator(model, X, features=["age"])

# No output here — this is a structural reference.

TreeExplainer is the fast SHAP variant for tree/boosting models; LIME and PDP treat the model as a queryable black box.

4 Why It Matters

In regulated fields — lending, healthcare, hiring, insurance — a decision must be explainable and auditable. People affected have a right to know why, and organisations must check models for bias and fairness. Interpretability also helps you debug: a feature dominating for the wrong reason often reveals leakage or a flawed pipeline.

🎯 Your Turn 1: SHAP Additivity

Fill in the blanks so the contributions sum to give the prediction. The expected output is in the comments.

# 🎯 YOUR TURN — verify the SHAP additivity property.
# SHAP contributions must sum to (prediction - baseline).

baseline = 0.40
contributions = {"a": 0.05, "b": -0.15, "c": 0.20}

total = sum(contributions.____())     # 👉 sum the .values() of the dict
prediction = baseline + ___           # 👉 add 'total' to the baseline
print("prediction:", round(prediction, 2))

# ✅ Expected output:
# prediction: 0.5

🎯 Your Turn 2: Biggest Driver

Find the feature that pushed a prediction up the most. Fill in the blank in the max() key.

# 🎯 YOUR TURN — find the feature that pushed a prediction up the MOST.
# Rank features by their (signed) contribution; biggest positive wins.

contributions = {"income": -0.10, "age": 0.05, "defaults": 0.25}

def biggest_driver_up(contribs):
    # max() by VALUE returns the feature with the largest positive push
    return max(contribs, key=lambda f: contribs[___])   # 👉 key on contribs[f]

print(biggest_driver_up(contributions))

# ✅ Expected output:
# defaults

🎯 Mini-Challenge: Explain a Prediction

Build a full SHAP-style explanation from baseline to prediction and name the biggest driver. Only a comment outline is provided.

# 🎯 MINI-CHALLENGE: explain a prediction with SHAP-style contributions
# Build the explanation and print it from baseline to prediction.
#
# 1. baseline = 0.20
# 2. contributions = {"hours": 0.15, "sleep": 0.05, "absences": -0.10}
# 3. prediction = baseline + sum of contributions
# 4. print() each feature with its contribution, then the final prediction
# 5. print() the single biggest driver (largest absolute contribution)
#
# ✅ Expected: prediction = 0.30, biggest driver = "hours"

# your code here

5 Common Errors (And How to Fix Them)

These are the traps when explaining models. Watch for them.

❌ Treating an explanation as ground truth

SHAP and LIME describe the model, not reality — a confident explanation of a wrong model is still wrong.

# ❌ "SHAP says age matters, so age causes the outcome" -> not causal

✅ Fix: read them as how the model behaves, and corroborate:

# SHAP shows the model RELIES on age; investigate before claiming cause

❌ Confusing global importance with a single decision

Global importance can't explain why one specific person was rejected.

model.feature_importances_   # ❌ global only, not per-prediction

✅ Fix: use local SHAP/LIME for one prediction:

explainer.shap_values(X[row])   # explains THIS prediction

❌ Using a black box when the law needs transparency

Sometimes a post-hoc explanation isn't enough and a directly interpretable model is required.

DeepNeuralNet()   # ❌ if regulation demands a readable model

✅ Fix: prefer an inherently interpretable model when mandated:

LogisticRegression()   # coefficients are directly explainable

📋 Quick Reference

🎉 Track Complete!

You can now explain a model's decisions with feature importance, SHAP values, LIME, and partial dependence plots, and you know when to choose a white-box model outright.

🎓 That wraps up the Classical Machine Learning track — from SVMs and clustering through boosting, forecasting, anomaly detection, and now interpretability. Well done!

Practice quiz

Continue this course

• Previous: Anomaly Detection
• Next: LLM Agents & Tool Use — Function/tool calling, the ReAct loop, planning and guardrails for agentic LLMs
• Quick reference: AI & Machine Learning cheat sheet
• From the blog: Neural Networks: An Introduction

Frequently asked questions

What is model interpretability and why does it matter?

Interpretability is the degree to which a human can understand why a model made a prediction. It matters because high-stakes decisions — loan approvals, medical diagnoses, hiring — need to be explained, audited for fairness, and trusted. A model that is accurate but unexplainable can be a legal and ethical liability.

What is the difference between a white-box and a black-box model?

White-box (interpretable) models like linear regression, logistic regression, and small decision trees let you read the decision directly from coefficients or splits. Black-box models like deep neural networks and large gradient-boosting ensembles are far more accurate on many problems but their internal reasoning is opaque — that's where SHAP and LIME come in.

What is feature importance and how is it different from a SHAP value?

Global feature importance ranks which features matter most across the whole model. A SHAP value is local and per-prediction: it tells you how much each feature pushed this particular prediction above or below the average. SHAP values for one prediction add up to the gap between that prediction and the model's baseline output, so they explain a single decision precisely.

How do SHAP and LIME work, and how do they differ?

SHAP is grounded in Shapley values from cooperative game theory: it fairly distributes a prediction among the features, with strong consistency guarantees, but can be slower. LIME builds a simple, interpretable model in the neighbourhood of a single prediction by perturbing the input and seeing how the output changes. SHAP is more rigorous and additive; LIME is faster and intuitive but only locally faithful.

What is a partial dependence plot?

A partial dependence plot (PDP) shows how the model's predicted outcome changes as you vary one feature across its range, averaging over all the other features. It reveals the overall shape of a feature's effect — for example, whether risk rises steadily with age or jumps at a threshold. It's a global view, complementing the local explanations from SHAP and LIME.

Are SHAP and LIME model-specific?

Both are mostly model-agnostic and post-hoc, meaning you apply them after training to almost any model. SHAP has fast specialised versions for tree models (TreeSHAP) and a general KernelSHAP for anything; LIME treats the model as a black box it queries. So you can use them on neural nets, boosting models, SVMs, and more.