SAM & Promptable Segmentation

SAM Promptable Segmentation Paradigm

  IMAGE ENCODER (Heavy ViT-H Backbone) ──► Image Embedding Tensor [1 x 256 x 64 x 64] (Computed ONCE!)
                                                                 │
  INTERACTIVE PROMPTS                                            │
  - Point Click (x, y) ──┐                                       │
  - Bounding Box       ├──► [ LIGHTWEIGHT PROMPT ENCODER ] ──────┼──► [ MASK DECODER (<50ms) ] ──► Mask
  - Text Prompt        ──┘                                       │

Architecture Components

1. Image Encoder: Heavy Vision Transformer (ViT-H / ViT-L) pre-trained with MAE (Masked Autoencoders). Takes $1024 \times 1024$ image input, generating a $64 \times 64 \times 256$ spatial feature embedding tensor.
2. Prompt Encoder: Embeds interactive user prompts:
   • Points / Bounding Boxes: Encoded using positional encodings + learned prompt type embeddings.
   • Text: Encoded using CLIP text transformer embeddings.
3. Mask Decoder: Lightweight Two-Way Transformer cross-attending prompt embeddings with image embedding maps, outputting vector masks and predicted IoU scores in $< 50\text{ms}$ execution time.

Resolving Prompt Ambiguity

A single point click is mathematically ambiguous:

  User Clicks on Shirt Button:
  - Mask Option 1 (Part):      Button
  - Mask Option 2 (Sub-Whole): Shirt
  - Mask Option 3 (Whole):     Whole Person

SAM solves ambiguity by predicting 3 candidate output masks simultaneously along with predicted IoU quality scores ($p_1, p_2, p_3$), allowing downstream applications to select the appropriate granularity.

The SA-1B Data Engine

Building the 1.1 Billion mask SA-1B dataset required a 3-stage data engine loop:

1. Assisted-Manual Stage: Model assists human annotators by generating candidate masks; humans refine boundaries.
2. Semi-Automatic Stage: Model automatically detects confident masks; human annotators label remaining un-segmented objects.
3. Fully Automatic Stage: Fully autonomous model generates dense grid masks across 11 Million images.

Say this out loud

Segment Anything Model (SAM) is a zero-shot promptable segmentation foundation model. It decouples a heavy ViT Image Encoder from a lightweight Mask Decoder. Running the image encoder once allows real-time interactive mask generation (<50ms) in browsers for point, box, or text prompts. Trained on 1.1 billion masks in SA-1B, SAM generalizes zero-shot across arbitrary unseen images.

Follow-ups to expect

• What is SAM 2 (Segment Anything in Images and Video - Meta 2024)? Extends SAM to real-time video tracking by adding a Memory Bank module (Memory Encoder, Memory Attention, Memory Bank) that stores object features across past video frames.
• How do you fine-tune SAM for specialized medical imaging (MedSAM)? Freeze the heavy ViT Image Encoder and train lightweight LoRA adapter layers on the Mask Decoder using medical dataset pairs (CT/MRI scans).