Conditional Independence

Core Definition & Factorization

Two variables $X$ and $Y$ are conditionally independent given $Z$ (written $X \perp Y \mid Z$) if and only if:

$$P(X, Y \mid Z) = P(X \mid Z) P(Y \mid Z)$$

Equivalently:

$$P(X \mid Y, Z) = P(X \mid Z) \quad \text{and} \quad P(Y \mid X, Z) = P(Y \mid Z)$$

Knowing $Y$ gives zero extra information about $X$ when $Z$ is already observed.

   Spurious Association (X <---> Y)        Conditioning on Z Breaks the Link!
              [ Z ]                                      [ Z ] (Observed)
             /     \                                    /     \
            ▼       ▼                                  ▼       ▼
          [ X ]   [ Y ]                              [ X ] ┴ [ Y ] (X ⊥ Y | Z)

Independence vs Conditional Independence

• Marginal Independence ($X \perp Y$): $P(X, Y) = P(X) P(Y)$. (Does NOT imply conditional independence).
• Conditional Independence ($X \perp Y \mid Z$): $P(X,Y|Z) = P(X|Z)P(Y|Z)$. (Does NOT imply marginal independence).

Classic Example: Common Cause (Fork)

• $X$: Stork population in a region.
• $Y$: Human birth rate.
• $Z$: Rural / Urban geographic setting.

$X$ and $Y$ are correlated marginally ($P(X, Y) \neq P(X)P(Y)$). But conditioned on location setting $Z$, $X \perp Y \mid Z$.

Applications in Machine Learning

1. Naive Bayes Classifier: Assumes all features $X_1, \dots, X_d$ are conditionally independent given class label $Y$:

$$P(X_1, \dots, X_d \mid Y) = \prod_{i=1}^d P(X_i \mid Y)$$

Converts exponential $O(K^d)$ parameter space into linear $O(d \cdot K)$ space!

2. Bayesian Networks & Markov Blankets: A node $X$ in a Directed Acyclic Graph (DAG) is conditionally independent of all non-descendants given its parents.
3. Causal Inference (d-separation): Blocking confounders $Z$ isolates true causal effects $X \to Y$.

Say this out loud

"Conditional independence X ⊥ Y | Z means P(X,Y|Z) = P(X|Z)P(Y|Z). Once Z is known, Y provides zero additional information about X. Spurious correlations between features often disappear when conditioning on a common cause Z. Naive Bayes relies on conditional independence of features given class Y to factorize high-dimensional joint likelihoods into simple 1D probability multiplications."

Follow-ups to expect

• What is the Explaining Away effect (V-structure / Collider)? In a collider DAG $X \to Z \leftarrow Y$, $X$ and $Y$ are marginally independent ($X \perp Y$), but conditioning on collider $Z$ makes $X$ and $Y$ dependent ($X \not\perp Y \mid Z$).
• What is a Markov Blanket? The set of nodes consisting of a target node's parents, children, and children's other parents. Conditioned on its Markov Blanket, a node becomes conditionally independent of all other nodes in the network.