Bias & Fairness in ML

Identifying and mitigating algorithmic bias across protected demographic groups in production machine learning.

🟡 intermediate5 min readethicsmust-know

Model Bias and Fairness evaluate whether machine learning systems produce equitable predictions across protected sub-groups (e.g. race, gender, age). Bias enters pipelines through historical data sampling, proxy features (e.g. ZIP code encoding race), and label bias. Key fairness criteria—Demographic Parity, Equalized Odds, and Predictive Parity—are mathematically incompatible (Impossibility Theorem of Fairness), requiring explicit product trade-offs across pre-processing, in-processing, and post-processing mitigation techniques.

Where Bias Enters the Pipeline

  Historical Human Bias ──► Representation Bias ──► Measurement / Label Bias ──► Proxy Feature Bias
  (Historical discrimination)  (Under-sampling group A) (Subjective human labels)   (ZIP code proxies race)

Fairness Through Unawareness (Myth): Removing protected columns (Gender, Race) fails because non-protected attributes (ZIP code, education, web history) act as high-capacity proxy features.

Three Core Mathematical Fairness Metrics

Given prediction $\hat{Y} \in {0, 1}$, target $Y \in {0, 1}$, and protected attribute $A \in {a, b}$:

1. Demographic Parity (Independence)

Positive prediction rate must be identical across groups, regardless of target distribution:

$$P(\hat{Y} = 1 | A = a) = P(\hat{Y} = 1 | A = b)$$

2. Equalized Odds (Separation)

True Positive Rate (Sensitivity) and False Positive Rate must be equal across groups:

$$P(\hat{Y} = 1 | Y = y, A = a) = P(\hat{Y} = 1 | Y = y, A = b), \quad \text{for } y \in {0, 1}$$

3. Predictive Parity (Sufficiency / Calibration)

Precision (Positive Predictive Value) must be equal across groups:

$$P(Y = 1 | \hat{Y} = 1, A = a) = P(Y = 1 | \hat{Y} = 1, A = b)$$

The Impossibility Theorem of Fairness (Kleinberg 2016)

Except in trivial cases (100% perfect model or identical base rates $P(Y=1|A=a) = P(Y=1|A=b)$), it is mathematically impossible to satisfy Demographic Parity, Equalized Odds, and Predictive Parity simultaneously.

You must choose which metric aligns with product and ethical goals!

Bias Mitigation Strategies

Pre-Processing: Re-weighting training samples, synthetic oversampling (SMOTE) for underrepresented groups, or Disparate Impact Remover.
In-Processing: Adversarial Debiasing (train an adversary to predict protected attribute $A$ from embeddings $z$, minimizing $L_{\text{task}} - \lambda L_{\text{adv}}$).
Post-Processing: Group-specific decision threshold adjustment (tuning $\tau_a \neq \tau_b$ to equalize TPR/FPR).

Say this out loud

"Model bias occurs when systems produce disparate impacts across protected groups. Simply dropping protected features fails because proxy attributes like ZIP code reconstruct demographic signals. The Impossibility Theorem proves that Demographic Parity, Equalized Odds, and Calibration cannot be satisfied simultaneously when base rates differ. We mitigate bias using pre-processing re-weighting, in-processing adversarial debiasing, or group-specific post-processing threshold tuning."

Follow-ups to expect

What is Disparate Impact (80% Rule)? A legal standard in hiring/credit: Disparate Impact occurs if the selection rate for a protected group is less than 80% (4/5ths) of the selection rate for the highest-performing group: $\frac{P(\hat{Y}=1 | A=minority)}{P(\hat{Y}=1 | A=majority)} < 0.80$.
How does Adversarial Debiasing work? An encoder extracts feature representation $z$. The primary task head predicts target $y$ from $z$, while an adversary network tries to predict protected attribute $A$ from $z$. Gradient reversal forces $z$ to discard all demographic information.

Check yourself

Question 1 of 3

Why does dropping protected attributes (e.g. removing 'Gender' or 'Race' columns) fail to prevent algorithmic model bias?

More in Responsible AI & Behavioural

See all →

Telling Your ML Project Story5 min Explainability: SHAP & LIME4 min Explaining a Model to a Non-Technical Exec5 min