Zero-shot, Few-shot & In-Context Learning

The Spectrum of Prompt-Based Task Learning

Before GPT-3, adapting an NLP model to a new task required downloading training datasets and fine-tuning parameter weights.

GPT-3 (Brown et al., 2020) demonstrated that Large Language Models possess In-Context Learning (ICL): the ability to learn new tasks directly from prompt context without updating a single model weight!

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. ZERO SHOT             │ 2. ONE SHOT              │ 3. FEW SHOT (K-SHOT)     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Task instructions ONLY.  │ Instructions + 1         │ Instructions + K         │
│ Zero demonstration       │ demonstration example.   │ demonstration examples.  │
│ examples provided.       │ Helps lock in format.    │ Maximum prompt accuracy! │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Zero-Shot Prompting

Asks the model to complete a task based purely on natural language instructions:

Classify the sentiment of this review as Positive or Negative:
Review: "The food was cold and service was slow."
Sentiment:

• Pros: Fast, minimal context window token usage.
• Cons: Struggles on complex custom schemas, domain-specific classification, or unusual formatting constraints.

2. Few-Shot Prompting (K-Shot Exemplars)

Provides $K$ concrete demonstration examples ($K = 2\text{--}5$) showing the exact input-to-output pattern expected:

Classify sentiment as Positive or Negative.

Review: "Great battery life and fast screen."
Sentiment: Positive

Review: "Broke on the first day, terrible quality."
Sentiment: Negative

Review: "Arrived on time and works as described."
Sentiment:

• Pros: Drastically improves formatting adherence, classification accuracy, and edge-case handling.
• Cons: Consumes context window tokens and increases API inference cost.

How In-Context Learning Works Inside Transformers

Why does providing 3 examples in a prompt change how a model processes the 4th query?

Research (von Oswald et al., 2023; Dai et al., 2022) revealed two mechanisms:

1. Implicit Gradient Descent: Transformer self-attention mechanisms operate on exemplar tokens, computing internal activation updates that mathematically mimic a step of implicit gradient descent during the forward pass!
2. Induction Heads: Specialized attention heads detect repeating patterns (Input A -> Output B) in the prompt and copy the mapped pattern when identical input structures recur.

Common Pitfalls in Few-Shot Prompting

1. Majority Class Bias: If your 5-shot prompt contains 4 Positive examples and 1 Negative example, the model becomes heavily biased toward outputting "Positive". Keep class ratios balanced.
2. Order Sensitivity: Changing the ordering of exemplars (e.g. swapping Example 1 and Example 3) can alter predictions by up to 20 percent.
3. Format Leakage: Incorrect formatting inside an exemplar (e.g. missing a trailing colon) will be faithfully copied by the model.

Say this out loud

Zero-Shot prompting provides task instructions alone. Few-Shot prompting adds K demonstration exemplars to the prompt context. In-Context Learning is an emergent property where self attention mechanisms infer task rules dynamically during forward passes without updating model weights. Few-Shot prompts require balanced class exemplars to avoid recency and majority biases.

Followups to expect

1. What is Many-Shot In-Context Learning? Long-context LLMs (Gemini 1.5 Pro, Claude 3.5) support 100-shot or 1000-shot prompting, filling context windows with hundreds of examples to rival fine-tuned model performance.
2. What is K-Nearest Neighbor Exemplar Selection? Instead of static few-shot examples, use a vector database to dynamically retrieve the $K$ most semantically similar past examples for each user query.