CLIP & Image–Text Alignment

The Dual-Encoder Contrastive Training Architecture

  BATCH OF N IMAGE-TEXT PAIRS: (Image_1, Text_1), (Image_2, Text_2), ..., (Image_N, Text_N)

  [ Image_1, Image_2, ..., Image_N ] ──► [ IMAGE ENCODER (ViT) ] ──► Normalized Image Embeddings I [N x D]
  [ Text_1,  Text_2,  ..., Text_N  ] ──► [ TEXT ENCODER (Trsf) ] ──► Normalized Text Embeddings  T [N x D]

                                          SIMILARITY MATRIX S = I · T^T [N x N]
                                          Text_1     Text_2     Text_3   ...  Text_N
                              Image_1   [ POSITIVE  Negative   Negative       Negative ]
                              Image_2   [ Negative  POSITIVE   Negative       Negative ]
                              Image_3   [ Negative  Negative   POSITIVE       Negative ]
                              ...
                              Image_N   [ Negative  Negative   Negative       POSITIVE ]

Mathematical Symmetric Loss Function

Given normalized image embeddings $\mathbf{I}_i$ ($|\mathbf{I}_i| = 1$) and text embeddings $\mathbf{T}_j$ ($|\mathbf{T}_j| = 1$):

Logit similarity matrix $S_{i,j} = (\mathbf{I}_i \cdot \mathbf{T}_j) \cdot e^\tau$, where $\tau$ is a learned temperature parameter.

Symmetric Cross-Entropy Loss:

$$\mathcal{L}{\text{Image}} = -\frac{1}{N} \sum{i=1}^N \log \frac{\exp(S_{i,i})}{\sum_{j=1}^N \exp(S_{i,j})}$$

$$\mathcal{L}{\text{Text}} = -\frac{1}{N} \sum{j=1}^N \log \frac{\exp(S_{j,j})}{\sum_{i=1}^N \exp(S_{i,j})}$$

$$\mathbf{\mathcal{L}{\text{CLIP}} = \frac{1}{2} \left( \mathcal{L}{\text{Image}} + \mathcal{L}_{\text{Text}} \right)}$$

Zero-Shot Classification via Text Prompting

To classify an image without training task-specific classification heads:

  1. Construct Class Prompts:  ["a photo of a dog", "a photo of a cat", "a photo of a car"]
  2. Encode Text Prompts:       T = TextEncoder(Prompts) ──► T [C x D]
  3. Encode Input Image:        i = ImageEncoder(Image)  ──► i [1 x D]
  4. Compute Cosine Scores:     Scores = Softmax( i · T^T / τ )
  5. Select Top Prediction:    "a photo of a dog" (Score = 0.94!)

Prompt Engineering in Zero-Shot CLIP

Passing raw class names like "dog" yields lower accuracy than contextualized prompts.

Using prompt templates like "a photo of a {class}, a type of animal." boosts zero-shot accuracy by +5% to +10% on ImageNet because CLIP pre-training text included full captions!

Say this out loud

CLIP unifies vision and natural language by training dual Image and Text Encoders on 400 million image-text pairs using symmetric in-batch contrastive loss. CLIP enables zero-shot classification by computing cosine similarity between an input image embedding and candidate text prompt embeddings ("a photo of a {class}"). CLIP embeddings power text-to-image models like Stable Diffusion and DALL-E 2.

Follow-ups to expect

• What is Open-Vocabulary Object Detection (OWL-ViT)? Combining CLIP text encoders with object detection heads to detect and ground arbitrary un-seen text descriptions in images without fixed class vocabularies.
• What is Vision-Language Alignment Bottleneck? CLIP aligns global image embeddings with global text captions. It struggles with fine-grained spatial grounding (e.g. distinguishing "red hat on blue car" vs "blue hat on red car").