Labelling: Weak Supervision & Annotation

The Training Data Bottleneck

In supervised machine learning, getting labeled data is often the hardest and most expensive step.

Manual human hand-labeling of 100,000 text documents or images takes months and costs tens of thousands of dollars.

To build training datasets rapidly, modern MLOps teams use Hybrid Labeling Strategies:

┌─────────────────────────────────────────────────────────────┐
│ 1. MANUAL ANNOTATION:   Human hand-labeling (High Quality, Expensive). │
│ 2. WEAK SUPERVISION:   Programmatic rules via Snorkel (Fast, Scalable).│
│ 3. PSEUDO LABELING:    Self-training model predictions (Semi-Supervised).│
│ 4. FOUNDATION MODELS:  Zero-shot LLM / VLM labeling (Distillation).    │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
                  High Quality Labeled Dataset!

1. Programmatic Weak Supervision (Snorkel)

Instead of humans hand-labeling every individual sample, domain experts write Labeling Functions (LFs) in Python:

# Example Snorkel Labeling Functions for Spam Detection
def check_spam_keywords(text):
  return SPAM if "click here now" in text.lower() else ABSTAIN


def check_short_url(text):
  return SPAM if "bit.ly" in text else ABSTAIN

Snorkel takes hundreds of noisy Labeling Functions, estimates their relative accuracies and correlations without ground truth data, and outputs a single Probabilistic Training Label for every sample!

2. Pseudo Labeling (Self-Training)

1. Train an initial machine learning model on a small seed set of 1,000 manually labeled samples.
2. Run the model over 100,000 unlabeled samples.
3. Select predictions with high confidence scores ($> 95%$) and treat them as true ground truth Pseudo Labels.
4. Combine seed labels and pseudo labels to retrain a larger model!

3. Foundation Model Distillation

Pass complex unlabeled text or images through a large foundation model (like GPT-4 or Claude) with a zero-shot prompt asking for classification labels.

Use the foundation model generated labels to train a much smaller, faster downstream model (like a BERT or ResNet classifier) for production serving.

Labeling Strategy Comparison

Say this out loud

Labeling strategies generate ground truth annotations for supervised models. Manual human annotation provides high quality labels but is expensive and slow. Weak supervision uses programmatic labeling functions and tools like Snorkel to generate probabilistic labels automatically. Pseudo labeling and LLM distillation leverage pre trained models to label large datasets efficiently.

Followups to expect

1. What is Noise Awareness in training on weak labels? Training downstream models using loss functions (like cross-entropy with soft probabilistic targets) that account for the estimated error probability of weak labels.
2. What is Active Learning in data labeling? Selecting the most informative or uncertain unlabeled samples for human experts to label, maximizing model accuracy gains per human annotation hour.