Video Understanding & Temporal Models

What is Video Understanding?

Processing single images involves 2D spatial dimensions: Height ($H$) $\times$ Width ($W$) $\times$ Channels ($C$).

Video adds a critical third dimension: Time ($T$ / Frames):

$$\text{Video Tensor} \in \mathbb{R}^{T \times H \times W \times C}$$

Video Understanding tasks (action recognition, video classification, event detection) must model two distinct signals simultaneously:

1. Spatial Appearance: What objects are present in individual frames?
2. Temporal Dynamics: How are objects moving across sequential time frames?

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. 3D CONVOLUTIONS (C3D) │ 2. TWO STREAM NETWORKS   │ 3. VIDEO TRANSFORMERS    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Inflates 2D kernels into │ Stream 1: RGB Frames     │ TimeSformer / Video Swin.│
│ 3D kernels (T x H x W).  │ Stream 2: Optical Flow.  │ Separable Space-Time     │
│ Captures motion joint.   │ Explicit motion vectors. │ Self-Attention blocks.   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. 3D Convolutional Networks (C3D & Inflated 3D / I3D)

Standard 2D convolution slides a $3 \times 3$ filter across spatial pixels.

3D Convolution slides a $3 \times 3 \times 3$ filter across Height, Width, and Temporal Frames simultaneously:

  2D Conv Kernel (3 x 3):        Slides across spatial pixels (H, W).
  3D Conv Kernel (3 x 3 x 3):    Slides across spatial pixels AND temporal frames (T, H, W).

Inflated 3D ConvNets (I3D - Carreira & Zisserman, 2017)

Takes successful pretrained 2D image backbones (like ResNet or Inception) and inflates their 2D $N \times N$ weights into 3D $N \times N \times N$ weights by repeating 2D weights along the time dimension and dividing by $N$.

This allows leverage of ImageNet pretraining for 3D video tasks!

2. Two-Stream Networks (Simonyan & Zisserman, 2014)

Instead of forcing a single network to learn motion implicitly, Two-Stream networks split spatial and temporal jobs:

  RGB Video Frame ──────► [ Spatial Stream 2D CNN ]  ───┐
                                                         ├──► Late Fusion ──► Action Prediction
  Optical Flow Field ───► [ Temporal Stream 2D CNN ] ───┘

• Spatial Stream: Processes raw RGB frames to recognize objects, scenes, and background context.
• Temporal Stream: Processes pre-computed Optical Flow vectors (pixel displacement direction and speed vectors between consecutive frames) to capture motion.

3. Video Transformers (TimeSformer & Video Swin)

Applying full $O(N^2)$ self-attention across all pixels in a 300 frame video causes catastrophic memory crashes.

TimeSformer (Bertasius et al., 2021) introduced Divided Space-Time Attention:

  Input Patch Tokens [T x H x W]
        │
        ├─► [ Temporal Self-Attention ] ──► Attends across TIME (Same pixel position across frames)
        │
        └─► [ Spatial Self-Attention ]  ──► Attends across SPACE (Pixels within the same frame)

Separating spatial attention from temporal attention reduces computation from $O((T \cdot H \cdot W)^2)$ down to $O(T^2 + (H \cdot W)^2)$, enabling long video context processing.

Say this out loud

Video Understanding processes 3D spatiotemporal tensors across Frames, Height, Width, and Channels. Architectures evolved from 3D Convolutions (C3D, I3D) and Two Stream Networks (RGB + Optical Flow) to Video Transformers like TimeSformer, which use Divided Space-Time Attention to process spatial and temporal dynamics separately.

Followups to expect

1. What is Optical Flow? A computer vision algorithm that calculates 2D displacement vectors $(dx, dy)$ for every pixel between consecutive video frames, capturing motion speed and direction.
2. What is Video Masked Autoencoding (VideoMAEE)? A self-supervised pretraining task for video transformers where 90 percent of video patches are masked out, forcing the model to reconstruct spatiotemporal video content from remaining patches.