Autoencoders
Compressing high dimensional inputs into low dimensional bottleneck vectors and reconstructing them.
What is an Autoencoder?
An Autoencoder is an unsupervised neural network that learns to compress input data into a compact summary vector, and then reconstruct the original input from that summary.
Input Data x (1024 dims) ──► [ ENCODER ] ──► Bottleneck Vector z (32 dims) ──► [ DECODER ] ──► Reconstructed x_hat (1024 dims)
(Compressed Representation!)
The network is trained by minimizing Reconstruction Loss:
$$\text{Loss} = | x - \hat{x} |^2$$
Because the goal is to make output $\hat{x}$ match input $x$, autoencoders require zero manual data labels (Self Supervised Learning).
Why the Bottleneck Layer is Essential
If hidden layers had the same size as the input, the network could trivially memorize an identity function ($y = x$), copying input pixels to output pixels without learning anything useful.
By forcing information through a narrow Bottleneck Layer ($z$) where dimension of $z$ is much smaller than input $x$, the network is forced to throw away noise and keep only the most important underlying structural patterns.
Core Applications
- Dimensionality Reduction: Non linear generalization of PCA. The bottleneck vector $z$ serves as a low dimensional embedding of high dimensional data.
- Anomaly Detection: Train an autoencoder exclusively on normal clean data (for example, normal credit card transactions). When an unusual fraud event passes through, the autoencoder fails to compress and reconstruct it properly, triggering a high reconstruction error spike.
- Denoising Autoencoders: Add artificial noise to input images ($x + \text{noise}$), while training the decoder to output the original clean image ($x$). This forces the network to learn robust underlying shapes rather than pixel values.
Say this out loud
An Autoencoder compresses input data through a narrow bottleneck layer into a low dimensional representation z, and reconstructs original data x_hat using a decoder. Unsupervised training minimizes reconstruction loss ||x - x_hat||^2. Autoencoders are widely used for non linear dimensionality reduction, image denoising, and anomaly detection based on reconstruction error spikes.
Followups to expect
- How does a Variational Autoencoder (VAE) differ from a standard Autoencoder? Standard autoencoders map inputs to discrete point vectors z. VAEs map inputs to continuous probability distributions (mean and variance vectors), allowing smooth generative sampling of new unseen data.
- Can Autoencoders be built with Convolutional layers? Yes, Convolutional Autoencoders use 2D conv layers in the encoder to compress image dimensions, and Transposed Conv layers in the decoder to upsample back to original image size.
Check yourself
What prevents a standard Autoencoder from simply learning an identity mapping that copies input x directly to output x_hat without learning meaningful features?