Generative vs Discriminative Models

The Fundamental Probabilistic Split

Every probabilistic machine learning model operates on inputs $X$ and labels $Y$.

The key distinction is how the model models the probability distributions:

┌──────────────────────────┬──────────────────────────┐
│ 1. DISCRIMINATIVE MODELS │ 2. GENERATIVE MODELS     │
├──────────────────────────┼──────────────────────────┤
│ Estimates P(Y | X).      │ Estimates P(X, Y) =      │
│ Learns DECISION BOUNDARY │ P(X | Y) * P(Y).         │
│ separating classes.      │ Learns HOW DATA IS MADE  │
│ Cannot generate data!    │ for each class.          │
│ Logistic Regression, SVM │ Naive Bayes, VAEs, GANs  │
└──────────────────────────┴──────────────────────────┘

1. Discriminative Models: "Where is the boundary?"

Discriminative models model the conditional probability $P(Y \mid X)$ directly.

They do not care how features $X$ were generated or what individual classes look like in isolation. They care only about drawing a boundary that separates Class 0 from Class 1.

  Discriminative Perception:
  "I don't know what makes a cat a cat. All I know is that if a picture has triangular ears and whiskers, it is 99% likely to be a cat rather than a dog!"

• Objective: Maximize conditional log-likelihood $\sum \log P(Y_i \mid X_i)$.
• Examples: Logistic Regression, Support Vector Machines (SVM), Decision Trees, BERT, ResNet.
• Strength: Typically achieves higher classification accuracy when training data is abundant because all model capacity is focused on the decision boundary.

2. Generative Models: "How was this data generated?"

Generative models model the joint probability distribution $P(X, Y)$.

Using Bayes' Rule, they model the likelihood of observing features $X$ given class $Y$ ($P(X \mid Y)$) alongside class prior $P(Y)$:

$$P(Y \mid X) = \frac{P(X \mid Y) P(Y)}{P(X)}$$

  Generative Perception:
  "I have learned a detailed statistical template for what a Cat looks like P(X | Cat) and what a Dog looks like P(X | Dog). To classify a new image, I compare which template matches better!"

• Examples: Naive Bayes, Linear Discriminant Analysis (LDA), Variational Autoencoders (VAEs), GANs, Diffusion Models, GPT (Causal LM).
• Strength: Can sample from $P(X \mid Y)$ to generate brand-new synthetic data points (images, audio, text). Handles missing input features cleanly.

Classical Pairing: Naive Bayes vs Logistic Regression

Say this out loud

Discriminative models estimate conditional probability P(Y | X) directly, learning decision boundaries to separate classes. Generative models estimate joint probability P(X,Y) = P(X | Y) P(Y), modeling how data features are generated for each class. Discriminative models excel at classification, while Generative models can synthesize brand new data points.

Followups to expect

1. Is GPT a Generative or Discriminative model? GPT is a Generative model. It models the joint distribution of sequence tokens $P(X_1, X_2, \dots, X_N)$, allowing it to sample new synthetic text tokens step by step.
2. Why does Logistic Regression outperform Naive Bayes on large datasets? Ng & Jordan (2001) proved that Naive Bayes reaches its asymptotic error threshold faster with less data, but Logistic Regression achieves a lower final error rate as dataset size approaches infinity.