Supervised, Unsupervised & Self-Supervised
The fundamental taxonomy of learning paradigms: from explicit labels to hidden patterns and self-supervised pretraining.
Comparison Matrix
Supervised Unsupervised Self-Supervised
Data: (X, Y) Data: (X) Data: (X) -> Auto-generated Y
Goal: Map X -> Y Goal: Discover P(X) / Structure Goal: Predict masked/future X
Examples: Linear, GBDT, ResNet Examples: k-Means, PCA, GMM Examples: GPT, BERT, MAE
| Dimension | Supervised | Unsupervised | Self-Supervised |
|---|---|---|---|
| Input Data | Labeled pairs (X, Y) | Unlabeled data (X) | Unlabeled raw data (X) |
| Objective | Minimize prediction loss L(f(X), Y) | Minimize reconstruction error / WCSS | Predict part of X from remaining X |
| Key Tasks | Classification, Regression, Object Detection | Clustering, Dimensionality Reduction, Anomaly Detection | LLM Pretraining, Vision Pretraining (ViT) |
| Data Cost | High (Human annotation required) | Low (Raw unannotated data) | Low (Leverages internet-scale data) |
| Human Supervision | High | Zero | Zero (Implicit self-generated targets) |
Self-Supervised Learning: The Foundation Model Engine
Self-supervised learning reformulates unsupervised data into a supervised task by creating proxy tasks:
- Autoregressive / Causal LM (GPT): Mask future text tokens; task model to predict next token
P(x_t | x_<t). - Masked Autoencoders (BERT / MAE): Mask 15% of text tokens or 75% of image patches; task model to reconstruct missing content.
- Contrastive Learning (SimCLR / CLIP): Pull positive augmented views of same image together in embedding space while pushing different images apart.
After self-supervised pretraining on massive data, models are fine-tuned on small supervised datasets via SFT / LoRA.
Say this out loud
"Supervised learning maps features X to human labels Y for classification and regression. Unsupervised learning discovers latent structures, clusters, and low-dimensional representations in unlabeled X. Self-supervised learning bridges both by creating automated supervisory targets from raw data—like next-token prediction in LLMs—enabling pretraining on internet-scale datasets before task-specific supervised fine-tuning."
Follow-ups to expect
- What is Weak Supervision (e.g. Snorkel)? Using programmatic heuristics, rules, and external knowledge bases to noisily label large datasets automatically, followed by generative modeling to estimate label reliability.
- How does Reinforcement Learning (RL) differ from Supervised Learning? Supervised learning gets explicit correct target answers y. RL receives delayed scalar rewards r from an environment based on sequential actions, balancing exploration and exploitation.
Check yourself
Which learning paradigm powers modern Large Language Models (LLMs) like GPT-4 and LLaMA during pretraining?