Explainability: SHAP & LIME
Opening the black box: how game theory guarantees mathematically sound feature attribution in ML.
Shapley values from game theory
Consider a predictive model as a cooperative game where features are players collaborating to produce a prediction (payout f(x)).
The Shapley Value ϕ_i of feature i is its average marginal contribution across all possible feature subsets S ⊆ F \ {i}:
ϕ_i = ∑_{S ⊆ F \ {i}} [ |S|! (|F| - |S| - 1)! / |F|! ] · [ f(S ∪ {i}) - f(S) ]
The 4 Mandatory Axioms
- Efficiency:
∑_{i=1}^M ϕ_i = f(x) - E[f(x)](attributions sum to difference from baseline). - Symmetry: If features
iandjcontribute identically to all subsets,ϕ_i = ϕ_j. - Dummy (Null player): If feature
ichanges no predictions,ϕ_i = 0. - Additivity: For ensemble
f + g,ϕ_i(f + g) = ϕ_i(f) + ϕ_i(g).
SHAP vs LIME
| Feature | SHAP | LIME |
|---|---|---|
| Theoretical Foundation | Game Theory (Shapley Values) | Local Linear Surrogate Modeling |
| Consistency & Fairness | Guaranteed by Axioms | No theoretical guarantees (sampling noise) |
| Local Sum Property | ∑ ϕ_i = f(x) - E[f(x)] | Does not sum to prediction |
| Computation Speed | Slow for KernelSHAP, Fast for TreeSHAP (O(TL D²)) | Fast (Local sampling around point) |
| Global Interpretability | Yes (mean absolute SHAP across samples) | No (Purely local explanations) |
Global vs Local Interpretability
- Local Explanation (Force Plot / Waterfall): Shows exact numerical contribution
+or-of each feature pushing a single customer's loan approval score away from base valueE[f(x)] = 0.65to outputf(x) = 0.88. - Global Explanation (Beeswarm Summary Plot): Combines local SHAP values across thousands of test samples. Y-axis ranks overall feature importance; X-axis shows SHAP value impact; color indicates feature value (red = high, blue = low).
Say this out loud
"SHAP is the gold standard for model explainability because it is mathematically grounded in Shapley game theory. Unlike GBDT gain or LIME, SHAP guarantees the Efficiency axiom — meaning local feature attributions add up exactly to the difference between the model's prediction and the expected base rate. For tree models, TreeSHAP computes exact Shapley values in polynomial time."
Follow-ups to expect
- Why can GBDT gain feature importance be misleading compared to SHAP? Gain importance is biased toward high-cardinality continuous features, doesn't reflect direction of effect (+ vs -), and changes depending on tree depth ordering.
- How do you handle correlated features in SHAP? Correlated features split attribution values between them, which can understate individual feature importance. Observational SHAP vs Interventional SHAP address conditional vs marginal feature sampling.
- Can SHAP be used for deep learning models? Yes, DeepSHAP combines SHAP with DeepLIFT backpropagation rules to efficiently approximate Shapley values for neural networks.
Check yourself
What core property guarantees that the sum of all SHAP feature attribution values for a prediction equals (Prediction - Expected Base Value)?