Anchor Boxes & Anchor-Free Detection

What are Anchor Boxes?

An Anchor Box is a pre-defined reference box placed at every grid cell of a feature map.

Instead of asking a neural network to guess a bounding box anywhere in a $1080 \text{p}$ image from scratch, we tile a grid of reference templates across the image:

  Grid Cell (Center x, y) ──► Tiles 3 Reference Anchor Templates:
                              - Template 1: Square (1:1 aspect ratio)   [100px x 100px]
                              - Template 2: Tall (1:2 aspect ratio)     [ 70px x 140px]
                              - Template 3: Wide (2:1 aspect ratio)     [140px x  70px]

  Reference Anchor Template ──► [ Model Predicts Offsets Δx, Δy, Δw, Δh ] ──► Final Refined Bounding Box

Bounding Box Offset Math

Given reference anchor box center $(x_a, y_a)$ and dimensions $(w_a, h_a)$, the model outputs 4 regression offsets $(t_x, t_y, t_w, t_h)$:

x_center = x_a + t_x * w_a y_center = y_a + t_y * h_a width = w_a * exp(t_w) height = h_a * exp(t_h)

Using exponential $\exp(t_w)$ guarantees that predicted width and height remain strictly positive.

Anchor-Based vs Anchor-Free Detectors

1. Anchor-Based (Faster R-CNN, YOLOv3 - v5): Uses pre-clustered anchor boxes. High recall, but requires tuning anchor scale hyperparameters via K-Means on dataset box shapes.
2. Anchor-Free (FCOS, CenterNet, YOLOv8): Predicts object center points and distances to 4 box boundaries (left, right, top, bottom) directly. Eliminates anchor hyperparameter tuning completely.

Say this out loud

Anchor boxes are predefined reference shape templates tiled across feature maps. Object detectors predict relative offset regressions (dx, dy, dw, dh) to deform anchor boxes toward ground-truth objects. While anchor boxes provide useful shape priors, modern detectors like YOLOv8 are Anchor-Free, predicting object centerpoints directly to avoid manual hyperparameter tuning.

Follow-ups to expect

• How do you select anchor box sizes for a new custom dataset? Run K-Means clustering on the bounding box widths and heights in your training set (using 1 - IoU as distance metric) to find the top K most common object shapes.
• What is ATSS (Adaptive Training Sample Selection)? An automated method for assigning positive and negative training anchors based on statistical IoU distribution characteristics rather than fixed manual thresholds.