Pose Estimation & Keypoints

What is Pose Estimation?

Pose Estimation is a computer vision task that identifies spatial locations of specific anatomical joints (Keypoints) on humans or objects:

• 2D Keypoints: $(x, y)$ coordinates in pixel space (e.g. 17 keypoints in COCO dataset: nose, eyes, shoulders, elbows, wrists, knees, ankles).
• 3D Keypoints: $(x, y, z)$ spatial coordinates in real-world meters.

  Input Image ──► [ POSE ESTIMATION MODEL ] ──► Skeleton Graph (Nodes: Joints, Edges: Limbs)

Applications include motion capture in gaming, athletic performance tracking, ergonomic safety monitoring, and physical therapy analysis.

Top-Down vs Bottom-Up Pipelines

┌──────────────────────────┬──────────────────────────┐
│ 1. TOP-DOWN APPROACH     │ 2. BOTTOM-UP APPROACH    │
├──────────────────────────┼──────────────────────────┤
│ Step 1: Detect Human     │ Step 1: Detect ALL image │
│ Bounding Boxes (YOLO).   │ keypoints in one pass.   │
│ Step 2: Predict keypoints│ Step 2: Group keypoints  │
│ for each cropped box.    │ into skeletons (OpenPose)│
│ Scaling: O(People Count) │ Scaling: O(1) Constant!  │
└──────────────────────────┴──────────────────────────┘

  TOP-DOWN PIPELINE (HRNet / AlphaPose):
  Full Image ──► [ Human Detector ] ──► Box 1 (Person A) ──► [ Keypoint Net ] ──► Skeleton A
                                     ──► Box 2 (Person B) ──► [ Keypoint Net ] ──► Skeleton B

  BOTTOM-UP PIPELINE (OpenPose):
  Full Image ──► [ Joint Heatmap Detector ] ──► All Wrists, Knees, Elbows
             ──► [ Part Affinity Fields ]   ──► Group Joints into Skeletons A and B!

• Choose Top-Down: When high precision keypoint accuracy is required for a small number of people (e.g. clinical rehabilitation).
• Choose Bottom-Up: When processing crowded scenes with dozens of people in real time (e.g. sports stadium camera feeds).

Heatmaps vs Direct Coordinate Regression

Should a neural network output raw coordinate numbers $(x = 245.2, y = 118.7)$ directly?

Direct coordinate regression performs poorly because loss gradients are unstable.

Modern networks output 2D Spatial Gaussian Heatmaps $H_k \in \mathbb{R}^{H \times W}$ for each keypoint $k$:

  Peak Probability (1.0) centered at true joint location (x, y)
  ──► Smooth 2D Gaussian falloff around the joint!

The keypoint position is extracted by finding the peak location $\arg\max_{(x,y)} H_k(x, y)$ or computing the center of mass.

High-Resolution Network (HRNet - Sun et al., 2019)

Traditional CNNs downsample feature maps to low resolutions to save memory.

HRNet maintains high-resolution spatial representations throughout the entire network while connecting multi-resolution sub-streams in parallel. This preserves precise spatial pixel locations critical for sub-pixel keypoint accuracy.

Say this out loud

Pose Estimation detects body joints and skeletal keypoint coordinates. Top Down approaches detect human bounding boxes first then predict keypoints per person. Bottom Up approaches (OpenPose) detect all keypoints image wide first, grouping them into individual skeletons using Part Affinity Fields. Networks output 2D Gaussian heatmaps to preserve spatial precision.

Followups to expect

1. What are Part Affinity Fields (PAFs) in OpenPose? 2D vector fields that encode the degree of association and direction along limb connections between adjacent keypoint joints.
2. What is Object Keypoint Similarity (OKS)? A metric measuring keypoint distance error normalized by person scale and joint specific visibility factors, used to calculate mAP for pose benchmarks.