Multimodal Models (VLMs)

Extending Transformers beyond text to process images, audio, video, and text in a unified token space.

🔴 advanced5 min readmultimodal

Multimodal Large Language Models (MLLMs - GPT-4o, Gemini, LLaVA, Claude 3.5 Sonnet) process multiple modalities (Images, Audio, Video, Text) simultaneously. Visual inputs are processed via Vision Encoders (ViT / CLIP), which partition images into patches, convert patches to dense visual tokens, and project them into the LLM's text embedding space using Linear or Cross-Attention Projection layers. Native multimodal architectures (GPT-4o, Gemini 1.5) process text, audio, and visual tokens end-to-end within a single unified Transformer backbone.

Vision Encoder + Projection Architecture (LLaVA Pattern)

  RGB Image (224x224) ──► [ Vision Transformer (ViT / CLIP) ] ──► Image Patch Vectors [196 × 1024]
                                                                          │
                                                                          ▼
                                                       [ Vision-Language Projection MLP ]
                                                       Projects 1024-d ──► 4096-d LLM Space
                                                                          │
  Text Tokens: ["Describe", "this", "image:"] ───────────────────────────┤
                                                                          ▼
                                                       Concat Visual + Text Tokens [200 × 4096]
                                                                          │
                                                                          ▼
                                                       [ Autoregressive LLM Backbone ]

How Vision Transformer (ViT) Tokenizes Images

Input image $I \in \mathbb{R}^{H \times W \times C}$ (e.g. $224 \times 224 \times 3$).
Divide image into grid of $N$ non-overlapping patches of size $P \times P$ (e.g. $16 \times 16$ pixels):

$$N = \frac{H \cdot W}{P^2} = \frac{224 \cdot 224}{16 \cdot 16} = 196 \text{ Patches}$$

Flatten each patch into vector $x_p \in \mathbb{R}^{P^2 C = 768}$.
Pass flattened vectors through Linear Projection Matrix $E$ + add Positional Embeddings $E_{pos}$:

$$z_0 = [x_p^1 E; x_p^2 E; \dots; x_p^N E] + E_{pos}$$

These 196 patch vectors act identically to 196 text word tokens inside the Transformer self-attention layers!

Multimodal Architectural Spectrum

Architecture Type	Mechanics	Models	Key Advantage
Stitched / Late-Fusion	Separate ViT + Projection Layer + Text LLM	LLaVA, BLIP-2, MiniGPT-4	Modular; easy to build using off-the-shelf open models
Native / Early-Fusion	Single unified Transformer processing text, vision, audio tokens end-to-end	GPT-4o, Gemini 1.5 Pro	Sub-300ms real-time audio/vision streams; zero modular loss
Cross-Attention Fusion	Interleaves Vision Attention layers inside LLM layers	Flamingo, IDEFICS	Efficient handling of arbitrary interleaved text/image documents

Say this out loud

"Multimodal models process vision, audio, and text in a shared token space. Vision Transformers (ViT) partition images into 16x16 pixel patches, projecting them into patch embeddings. A Projection MLP maps visual vectors into the LLM's text embedding space, concatenating image and text tokens for unified self-attention."

Follow-ups to expect

How do Multimodal Models process high-resolution images? High-res images (e.g. 1080p) are split into dynamic grids of 448x448 tiles + 1 global thumbnail image, generating ~1,000+ vision tokens to preserve fine-grained text and UI details.
What is Audio Tokenization (SoundStream / EnCodec)? Converts raw continuous audio waveforms into discrete acoustic neural tokens (codebook indices) using Vector Quantization (VQ-VAE), allowing LLMs to process speech natively.

Check yourself

Question 1 of 3

How does a Vision Transformer (ViT) convert a 224x224 RGB image into tokens for an LLM embedding space?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min