IoU & Non-Max Suppression

1. Intersection over Union (IoU)

IoU measures spatial overlap accuracy between two bounding boxes:

IoU = Area of Overlap / Area of Union = Area( A n B ) / Area( A u B )

  BOX A (Prediction) and BOX B (Ground Truth):
  ┌───────────┐
  │   Box A   │
  │     ┌─────┼──────┐          Overlap Area (A ∩ B)
  │     │ A∩B │      │  ──►   ───────────────────────  =  IoU Score ∈ [0.0, 1.0]
  └─────┼─────┘      │          Union Area (A ∪ B)
        │   Box B    │
        └────────────┘

• IoU = 1.0: Perfect match (100% overlap).
• IoU >= 0.5: Standard threshold for declaring a prediction a "True Positive" hit.
• IoU = 0.0: Disjoint boxes (zero overlap).

2. Non-Maximum Suppression (NMS)

Dense object detectors predict dozens of overlapping boxes around a single physical object. NMS filters redundant candidates:

  RAW DETECTOR OUTPUT (Many Redundant Boxes):       AFTER NMS POST-PROCESSING:
       ┌──────┐ ┌──────┐ ┌──────┐                        ┌──────┐
       │ Box1 │ │ Box2 │ │ Box3 │    ──────────►         │ Box1 │ (Highest Score = 0.95)
       └──────┘ └──────┘ └──────┘                        └──────┘
       (Scores: 0.95, 0.88, 0.72)                        (Box2 & Box3 Suppressed!)

NMS Algorithm Steps

1. Filter out all predicted boxes with confidence scores below minimum threshold (e.g. score < 0.20).
2. Sort remaining candidate boxes by confidence score in descending order.
3. Select box $B_{\text{max}}$ with highest score and add it to final output set.
4. Compute IoU between $B_{\text{max}}$ and all remaining candidate boxes.
5. Suppress (Delete) any candidate box where $\text{IoU}(B_{\text{max}}, B_i) > \text{NMS}{\text{threshold}}$ (typically $\text{NMS}{\text{threshold}} = 0.45$).
6. Repeat steps 3–5 for remaining candidate boxes until set is empty.

Soft-NMS (Handling Crowded Scenes)

In crowded scenes (e.g. horses running side-by-side), Hard-NMS deletes real detection boxes because true objects overlap heavily.

Soft-NMS decays confidence scores continuously based on overlap rather than hard-deleting:

$$\text{Score}_i = \text{Score}i \cdot \exp\left( -\frac{\text{IoU}(B{\text{max}}, B_i)^2}{\sigma} \right)$$

Higher overlap reduces the score proportionally, allowing true secondary detections to survive if their feature evidence remains strong!

Say this out loud

IoU measures box overlap area divided by total union area. Non-Maximum Suppression (NMS) cleans up duplicate bounding box predictions around the same object by selecting the highest-confidence box and deleting surrounding candidates with high IoU overlap. Soft-NMS continuously decays confidence scores instead of hard-deleting boxes, improving detection accuracy in crowded scenes.

Follow-ups to expect

• What is Fast-NMS (YOLACT)? A GPU-parallelized version of NMS that computes pairwise IoU matrix across all N boxes simultaneously in parallel, performing matrix masking instead of sequential iterative loops.
• Is NMS differentiable? No, standard NMS operations (sorting and thresholding) are non-differentiable step functions. End-to-end detectors like DETR avoid NMS using bipartite matching loss.