Semi-Supervised Learning

The Label Scarcity Problem

In real world applications:

• Unlabelled Data: Abundant and cheap (millions of web images, medical scans, or text documents).
• Labelled Data: Scarce and expensive (requires human experts or medical doctors to annotate).

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ SUPERVISED LEARNING      │ UNSUPERVISED LEARNING    │ SEMI SUPERVISED LEARNING │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ 100% Labelled Data.      │ 0% Labelled Data.        │ 5% Labelled Data +       │
│ Expensive to scale.      │ Learns patterns/clusters.│ 95% Unlabelled Data!     │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Semi Supervised Learning leverages large pools of unlabelled data to shape decision boundaries, drastically cutting human annotation requirements.

  Labelled Samples (Small Circles) + Unlabelled Pool (Large Dots)
  ──► Unlabelled data reveals underlying CLUSTER MANIFOLD structure!
  ──► Decision boundary shifts out of dense cluster regions into empty space!

Core Semi-Supervised Paradigms

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. PSEUDO LABELING       │ 2. CONSISTENCY REGULAR   │ 3. PRETRAIN + FINE TUNE  │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Model assigns labels to  │ Forces predictions to match│ Self-supervised pretrain │
│ unlabelled data; uses    │ under data augmentations │ on unlabelled data (BERT,│
│ confident samples.       │ (FixMatch / UDA).        │ SimCLR); fine-tune small.│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Pseudo Labeling (Self-Training)

1. Train an initial model $M$ on small labelled dataset $D_{\text{labeled}}$.
2. Predict probabilities on unlabelled pool $D_{\text{unlabeled}}$.
3. Filter high-confidence predictions (e.g. predicted probability $p > 0.95$).
4. Add these high confidence samples with their Pseudo-Labels into the training set.
5. Retrain model $M$ on combined data and repeat!

2. Consistency Regularization (FixMatch - Sohn et al., 2020)

Applies the Smoothness Assumption: small perturbations to an input should not change model predictions.

  Unlabelled Image X
        │
        ├─► [ Weak Augmentation (Flip/Shift) ]   ──► Model Pred P(y|X) ──► Pseudo Label (p > 0.95)
        │                                                                      │
        └─► [ Strong Augmentation (Cutout/Color) ] ──► Model Pred Q(y|X) ◄─────┘ (Calculate Cross Entropy Loss!)

FixMatch uses weak augmentations to generate pseudo-label targets, and forces the model to match those predictions when given strongly augmented versions of the exact same image!

3. Self-Supervised Pretraining + Fine Tuning (The Modern Standard)

Pretrain a massive backbone model on unlabelled data using Self-Supervised objectives (BERT Masked LM, GPT Causal LM, SimCLR Contrastive Vision).

The model learns rich general feature representations. Then, fine-tune on a small labelled dataset using a tiny fraction of labels!

Say this out loud

Semi Supervised Learning combines a small labelled dataset with a large unlabelled data pool. Pseudo Labeling uses confident model predictions on unlabelled data as pseudo ground truth targets. Consistency Regularization (FixMatch) forces model predictions to remain unchanged across augmented versions of unlabelled inputs, utilizing unlabelled data structure to refine decision boundaries.

Followups to expect

1. What is Confirmation Bias in Pseudo Labeling? If the initial model assigns an incorrect pseudo label with high confidence, retraining on that incorrect label reinforces the error, causing performance degradation.
2. What is the Low-Density Separation Assumption? The decision boundary between classes should pass through low-density regions in feature space rather than cutting through dense clusters of unlabelled data points.