Naive Bayes

The Naive Bayes Derivation

By Bayes' Theorem:

$$P(y \mid x_1, \dots, x_d) = \frac{P(y) P(x_1, \dots, x_d \mid y)}{P(x_1, \dots, x_d)}$$

Under the Conditional Independence Assumption:

$$P(x_1, \dots, x_d \mid y) = \prod_{i=1}^d P(x_i \mid y)$$

Dropping denominator $P(X)$ (constant for all classes $y$):

$$\hat{y} = \arg\max_y P(y) \prod_{i=1}^d P(x_i \mid y)$$

In log-space (to avoid floating-point underflow):

$$\hat{y} = \arg\max_y \left[ \ln P(y) + \sum_{i=1}^d \ln P(x_i \mid y) \right]$$

Three Core Naive Bayes Variants

1. Gaussian Naive Bayes: Continuous features $x_i \sim \mathcal{N}(\mu_{y,i}, \sigma_{y,i}^2)$. Uses Gaussian likelihood.
2. Multinomial Naive Bayes: Text classification with word frequency counts $x_i \in {0, 1, 2, \dots}$.
3. Bernoulli Naive Bayes: Binary features $x_i \in {0, 1}$ (explicitly penalizes absence of expected words).

Laplace (Additive) Smoothing

Unsmoothed feature likelihood:

$$P(x_i \mid y) = \frac{N_{y, i}}{N_y}$$

If word $i$ never appeared in training class $y$, $N_{y,i} = 0 \implies P(x_i \mid y) = 0$.

Laplace Additive Smoothing ($\alpha = 1$):

$$P(x_i \mid y) = \frac{N_{y, i} + \alpha}{N_y + \alpha |V|}$$

Where $|V|$ is the total vocabulary size.

Say this out loud

"Naive Bayes classifies samples by applying Bayes' Theorem under the conditional independence assumption P(x_1..x_d|y) = ∏ P(x_i|y). This turns high-dimensional joint estimation into fast 1D probability multiplications. We use Multinomial Naive Bayes for word counts and apply Laplace smoothing α = 1 to prevent unobserved words from zeroing out class probabilities."

Follow-ups to expect

• Why does Naive Bayes perform well even when conditional independence is violated? Classification relies on the argmax decision boundary (which class probability is highest), not exact probability calibration. Even if joint probabilities are inaccurate due to feature correlations, the ranking order of classes often remains correct.
• How does Naive Bayes handle zero-frequency continuous features? Gaussian Naive Bayes uses smoothing variance $\epsilon$ added to calculated feature variance $\sigma_{y,i}^2 + \epsilon$ to avoid division by zero.