Semantic vs Instance Segmentation

The Three Pillars of Image Segmentation

  INPUT IMAGE                  SEMANTIC SEGMENTATION           INSTANCE SEGMENTATION           PANOPTIC SEGMENTATION
  (Two people on a road)       (Class per pixel)               (Mask per individual object)   (Unified Stuff + Things)
  ┌──────────────────────┐     ┌──────────────────────┐        ┌──────────────────────┐        ┌──────────────────────┐
  │ [Person1] [Person2]  │     │ [Person]   [Person]  │        │ [Person1]  [Person2] │        │ [Person1]  [Person2] │
  │    (Road)  (Sky)     │     │    (Road)   (Sky)    │        │    (No Road) (No Sky)│        │    (Road)   (Sky)    │
  └──────────────────────┘     └──────────────────────┘        └──────────────────────┘        └──────────────────────┘
  All person pixels merged!    Individual masks!               Unified scene mask!

1. Semantic Segmentation

Assigns a class label $c \in {1, \dots, C}$ to every single pixel $(x, y)$ in the image.

It does NOT differentiate between multiple instances of the same class. Three cars parked in a row are merged into a single contiguous "car" region.

• Output tensor shape: $H \times W \times C$ (Spatial height $\times$ width $\times$ number of classes).
• Primary Loss: Pixel-wise Cross-Entropy Loss or Dice Loss.

2. Instance Segmentation (Mask R-CNN)

Detects and segments individual countable objects ("Things").

Mask R-CNN adds a small Fully Convolutional Network (FCN) branch to Faster R-CNN to predict a binary $m \times m$ pixel mask for each detected bounding box:

  RoI Align ──► [ Feature Crop ] ──┬──► [ Class Head ] ──► Label: "Person"
                                   ├──► [ Box Head ]   ──► Bounding Box [x, y, w, h]
                                   └──► [ Mask Head ]  ──► 28x28 Binary Mask

RoIAlign vs RoIPool

Standard RoIPool quantized region coordinates to integers, causing pixel misalignment.

RoIAlign uses bilinear interpolation to extract exact continuous spatial features without rounding errors, enabling precise pixel-level mask predictions.

3. Panoptic Segmentation

Combines the strengths of both worlds:

• Stuff: Uncountable background regions with no distinct boundaries (sky, road, ocean, sand).
• Things: Countable objects with distinct shapes (people, cars, animals, chairs).

Panoptic segmentation outputs a non-overlapping pixel map where every pixel receives a pair $(C_i, I_i)$: Class Category $C_i$ and Instance ID $I_i$.

Say this out loud

Semantic segmentation labels every pixel by category without separating individual objects. Instance segmentation detects and masks distinct object instances (Person 1 vs Person 2). Panoptic segmentation unifies both into a single complete scene understanding map, combining background stuff categories (road, sky) with individual object things (cars, people).

Follow-ups to expect

• What is Dice Loss / F1 Loss in segmentation? A loss function based on the Sørensen–Dice coefficient $L_{\text{Dice}} = 1 - \frac{2 |Y \cap \hat{Y}|}{|Y| + |\hat{Y}|}$, optimizing spatial mask overlap directly on imbalanced foreground-background images.
• What is Mask2Former? A unified Transformer architecture that solves Semantic, Instance, and Panoptic segmentation using a single Masked-Attention Query framework without task-specific architectural changes.