Depthwise Separable Convolutions

The Efficiency Problem in Standard Convolution

A standard 2D Convolution performs two jobs simultaneously:

1. Spatial Filtering: Looking at $D_K \times D_K$ pixel neighbors.
2. Channel Combination: Combining features across all $M$ input channels to produce $N$ output channels.

Computational Cost for standard 3x3 convolution:

Cost_Standard = H * W * M * N * D_K * D_K

For 512 input channels, 512 output channels, and 3x3 filters on a 14x14 feature map:

Cost_Standard = 14 * 14 * 512 * 512 * 3 * 3 = 235,929,600 FLOPs!

This is too expensive for mobile phones, drones, and edge devices.

The Depthwise Separable Solution

Depthwise Separable Convolution factorizes standard convolution into two independent steps:

  1. DEPTHWISE CONVOLUTION (Spatial Only)       2. POINTWISE CONVOLUTION (Channels Only)
  Apply ONE 3x3 filter per channel.             Apply 1x1 convolutions across channels.
  Filters spatial patterns independently.       Combines channel information together.
  ┌───┬───┬───┐                                 ┌───┐
  │ . │ . │ . │  Channel 1 ONLY                 │ 1 │  Blends 512 channels
  └───┴───┴───┘                                 └───┘  into N outputs

Step 1: Depthwise Convolution

Applies a single $3 \times 3$ filter to each input channel individually. Zero channel mixing happens here.

Cost_Depthwise = H * W * M * D_K * D_K

Step 2: Pointwise Convolution

Applies a $1 \times 1$ convolution across channels to blend features into $N$ output channels. Zero spatial neighbor mixing happens here.

Cost_Pointwise = H * W * M * N * 1 * 1

Total Combined Cost = Cost_Depthwise + Cost_Pointwise

Comparing the Computation Savings

Ratio of Depthwise Separable Cost to Standard Cost:

$$\text{Savings Ratio} = \frac{H \cdot W \cdot M \cdot D_K^2 + H \cdot W \cdot M \cdot N}{H \cdot W \cdot M \cdot N \cdot D_K^2} = \frac{1}{N} + \frac{1}{D_K^2}$$

For $3 \times 3$ filters ($D_K = 3$):

$$\text{Savings Ratio} \approx \frac{1}{9} \approx 11%$$

You get an 8 to 9 times reduction in total parameters and FLOPs (89 percent compute savings!) with almost zero drop in top 1 accuracy.

Say this out loud

Depthwise Separable Convolution splits standard convolution into spatial Depthwise filtering per channel and channel blending via 1x1 Pointwise convolutions. Decoupling spatial filtering from channel mixing cuts compute and parameter counts by 8 to 9 times, enabling real time computer vision models like MobileNet to run on smartphones.

Followups to expect

1. What is an Inverted Residual Block in MobileNetV2? Expands channels with 1x1 conv, applies 3x3 Depthwise conv, and shrinks channels back with 1x1 conv, using linear bottlenecks to prevent information loss.
2. How is 1x1 convolution used in general deep learning? Used for dimensionality reduction (shrinking channel count), channel expansion, and adding cheap non linearities without changing spatial height or width.