Interpretable Models vs Post-Hoc Explanations

The Great Interpretability Dilemma

When designing a machine learning system for high stakes decisions (medical diagnosis, criminal sentencing, credit lending), developers face a choice:

Option A: INHERENTLY INTERPRETABLE (Linear Reg, Decision Trees, EBMs)
          Fully transparent, exact auditability, zero approximation errors!

Option B: BLACK BOX + POST-HOC EXPLANATION (Deep Nets / XGBoost + SHAP)
          High predictive capacity, explained post-hoc via local approximations.

In a landmark 2019 paper, Cynthia Rudin argued forcefully:

"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead."

Inherently Interpretable Models (Interpretable by Design)

Interpretable by design models expose their exact decision mechanics directly:

• Linear / Logistic Regression: Coefficients state exact marginal weight impacts ($\beta_j$).
• Short Decision Trees: Tree rules can be printed and audited by human domain experts.
• Generalized Additive Models (GAMs / EBMs): Model outputs as sums of single feature shape functions:

$$g(E[y]) = \beta_0 + f_1(x_1) + f_2(x_2) + \dots + f_p(x_p)$$

Explainable Boosting Machines (EBMs / InterpretML)

Microsoft's EBM is a modern GAM that trains gradient boosted decision trees on one feature at a time. It matches the accuracy of Random Forests on tabular data while remaining $100%$ inherently interpretable via 1D shape plots!

EBM 1D Shape Plot for Feature 'Age':
Contribution Score
+2.0 ┼                 ┌──────┐
 0.0 ┼─────────────────┘      └────────►
-2.0 ┴─────────────────────────────────►
     0       18       65      80  (Age)

The Pitfalls of Post-Hoc Explanations

Post-hoc explanation methods (SHAP, LIME) attempt to explain black box models after training:

1. Unfaithful Approximations: LIME fits a local linear model around a prediction. If the true black box boundary is non-linear, the explanation can be unfaithful to true model mechanics.
2. Adversarial Manipulation: Attackers can design black box models that produce benign SHAP explanations while harboring hidden discriminatory logic.
3. Explaining Errors: An explanation explains what the model predicted, not what is true in reality.

Say this out loud

Inherently interpretable models like GAMs and EBMs expose exact decision mechanics directly, avoiding approximation errors. Post-hoc explanation methods like SHAP and LIME approximate complex black box models after training, but can suffer from unfaithful local approximations. High stakes decisions should prioritize inherently interpretable models.

Followups to expect

1. What is the Accuracy vs Interpretability Trade-off Myth? The common belief that complex black box models always outperform simple models. On structured tabular datasets, GAMs and EBMs often match XGBoost accuracy while remaining fully interpretable.
2. What is a Surrogate Model? An inherently interpretable model (like a decision tree) trained to mimic the predictions of a complex black box model to approximate its global behavior.