Simpson’s Paradox

What is Simpson's Paradox?

Simpson's Paradox is a famous statistical phenomenon where an observed trend across aggregated data completely reverses direction when the data is split into underlying sub-groups.

  AGGREGATE LEVEL RESULT:   Treatment A appears BETTER than Treatment B!
                                       ▼ (Disaggregate into Sub-groups!)
  SUB-GROUP LEVEL RESULT:   Treatment B is BETTER than Treatment A in Group 1!
                            Treatment B is BETTER than Treatment A in Group 2!

This counter intuitive reversal causes massive misinterpretations in A/B testing, medical studies, and hiring analytics.

Famous Real-World Example: UC Berkeley Admissions (1973)

UC Berkeley was sued for gender bias because overall admission data showed men were admitted at a significantly higher rate than women:

  AGGREGATE DATA:
  - Male Applicants:   44% Admitted
  - Female Applicants: 35% Admitted  <-- (Appears to show bias against women!)

However, when researchers disaggregated data by Department:

  DEPARTMENT LEVEL BREAKDOWN:
  - Dept A: Men 62% Admitted | Women 82% Admitted (Women Higher!)
  - Dept B: Men 63% Admitted | Women 68% Admitted (Women Higher!)
  - Dept C: Men 37% Admitted | Women 34% Admitted (Equal)

In 4 out of 6 major departments, women were admitted at higher rates than men!

Why Did the Aggregate Data Flip?

Women applied in large numbers to competitive departments with tiny acceptance rates (like English and Psychology, 10% admission rate).

Men applied in large numbers to easy engineering departments with huge acceptance rates (60% admission rate).

The lurking Confounding Variable was Department Selectivity!

                       DEPARTMENT SELECTIVITY
                       (Confounding Variable)
                       ╱                    ╲
                      ▼                      ▼
  GENDER APPLICANT RATIO ──────────────► ADMISSION RATE

How to Resolve Simpson's Paradox

When encountering opposite trends in aggregate vs disaggregated data, which result is correct?

You cannot decide using raw statistics alone. You must construct a Causal Directed Acyclic Graph (DAG):

1. If the variable is a CONFOUNDER (like Department Choice): You MUST disaggregate data and evaluate subgroup results to eliminate confounding bias.
2. If the variable is a MEDIATOR (a step in the causal chain): You MUST aggregate data to measure total causal effect.

Say this out loud

Simpson's Paradox occurs when an overall statistical trend in aggregated data reverses direction when split into sub populations. It is caused by uncontrolled confounding variables that influence both group allocation and outcome metrics. Resolving the paradox requires causal DAG modeling to determine whether to aggregate or disaggregate data based on real world causal structures.

Followups to expect

1. How does Simpson's Paradox ruin A/B tests? If an A/B test allocates 80 percent of mobile users to Variant A and 80 percent of desktop users to Variant B, conversion rates can flip in aggregate due to device conversion rate differences.
2. What is Berkson's Paradox? A related selection bias phenomenon where two independent variables appear negatively correlated within a specialized sampled population (like hospital patients or university admissions).