Brier Score & Proper Scoring Rules
Evaluating probability accuracy and calibration simultaneously using Mean Squared Error over binary predictions.
What is the Brier Score?
When a weather forecasting model predicts a $70%$ chance of rain, how do we evaluate whether that single probability number was accurate after rain occurs ($y = 1$)?
The Brier Score (Brier, 1950) measures probability accuracy by calculating the Mean Squared Error between predicted probabilities $\hat{p}_i$ and actual binary outcomes $y_i \in {0, 1}$:
$$\text{Brier Score} = \frac{1}{N} \sum_{i=1}^N (\hat{p}_i - y_i)^2$$
Perfect Probabilistic Predictions ──► Brier Score = 0.00
Uninformative Baseline (p = 0.50) ──► Brier Score = 0.25
Completely Wrong Predictions ──► Brier Score = 1.00
Lower Brier Scores represent superior probability accuracy.
Strictly Proper Scoring Rules
The Brier Score and Log Loss are examples of Strictly Proper Scoring Rules.
A scoring rule is Proper if the forecaster minimizes expected loss if and only if they state their true honest probability distribution.
If an evaluation metric is not proper, models can game the metric by outputting extreme over-confident numbers ($0.0$ or $1.0$) rather than honest probabilities.
Murphy Decomposition of Brier Score
Brier Score decomposes mathematically into three distinct components:
$$\text{Brier Score} = \text{Reliability} - \text{Resolution} + \text{Uncertainty}$$
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. RELIABILITY │ 2. RESOLUTION │ 3. UNCERTAINTY │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Measures Calibration │ Measures Discrimination │ Measures inherent noise │
│ Error! Distance between │ Power. How much predictions│ in the true target data. │
│ predictions and empirical│ diverge from overall mean│ Out of model control! │
│ frequencies. Want ZERO! │ base rate. Want HIGH! │ │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
- Reliability: How well calibrated are probabilities? (Lower is better).
- Resolution: How well does the model separate distinct outcome risks? (Higher is better).
- Uncertainty: Inherent variance of the domain.
Brier Score vs Log Loss
| Metric | Formula | Outlier Sensitivity |
|---|---|---|
| Brier Score | Mean Squared Error $(\hat{p} - y)^2$ | Bounded error between $0.0$ and $1.0$ |
| Log Loss | Binary Cross-Entropy $-y\log(\hat{p})$ | Infinitely penalizes extreme wrong predictions |
Log Loss penalizes confident wrong predictions (predicting $0.999$ when outcome is $0$) with infinite loss, whereas Brier Score bounds the maximum penalty to $1.0$.
Say this out loud
The Brier Score measures probability accuracy using Mean Squared Error between predicted probabilities and binary outcomes. Lower scores indicate superior performance. As a Strictly Proper Scoring Rule, it incentivizes models to output honest probabilities. Brier Score decomposes into Reliability calibration error, Resolution discrimination power, and inherent task Uncertainty.
Followups to expect
- What is Brier Skill Score (BSS)? A normalized metric comparing a model Brier score against a simple climatological or historical base rate forecast: $\text{BSS} = 1 - (\text{BS}{\text{model}} / \text{BS}{\text{baseline}})$.
- Can Brier Score be extended to multi-class problems? Yes, Multi-Class Brier Score sums squared errors across all class probability components for each sample instance.
Check yourself
What mathematical operation defines the Brier Score for binary probabilistic predictions?