OCR Pipelines

Industrial 2-Stage OCR Pipeline

  INPUT SCENE IMAGE            STAGE 1: TEXT DETECTION           STAGE 2: TEXT RECOGNITION
  (Receipt / Street Sign)      (DBNet / EAST / PaddleOCR)        (CRNN + CTC / TrOCR)
  ┌──────────────────────┐     ┌──────────────────────┐          ┌─────────────────────────┐
  │  [ RECEIPT HEADER ]  │ ──► │ Crop Polygons:       │ ──►      │ Crop 1: "WALMART STORE" │
  │  [ ITEM 1: $12.99 ]  │     │ - Polygon 1 [x,y..]  │          │ Crop 2: "TOTAL $12.99"  │
  └──────────────────────┘     │ - Polygon 2 [x,y..]  │          └─────────────────────────┘
                               └──────────────────────┘          Output: Clean Text Json!

Stage 1: Text Detection (DBNet / Real-Time Detection)

Unlike standard object detection, scene text appears at arbitrary angles, curved shapes, and extreme aspect ratios.

DBNet (Real-time Scene Text Detection with Differentiable Binarization): Predicts probability maps and adaptive threshold maps, using a differentiable step function to output precise text boundary polygons directly.

Stage 2: Text Recognition (CRNN + CTC Loss)

Given a cropped text line image:

  Text Line Crop Image ──► [ CNN Feature Extractor ] ──► [ Map-to-Sequence ] ──► [ Bidirectional LSTM ] ──► CTC Loss

Connectionist Temporal Classification (CTC) Loss

In text recognition, annotators provide ground-truth text "CAT", but do NOT label where letter 'C' ends and 'A' starts in the image pixels.

CTC introduces a special Blank Token ($\epsilon$) and collapses repeated characters:

$$\text{Output Sequence: } \text{"C - C - - A - - T - T"} \xrightarrow{\text{Collapse Repeats & Blanks}} \text{"CAT"}$$

CTC computes the sum of probabilities over all valid aligned sequences using dynamic programming!

Modern Vision Transformer OCR (TrOCR)

TrOCR (Microsoft, 2021) replaces CNN-RNN-CTC pipelines with a pure Transformer Encoder-Decoder:

  Text Crop Image ──► [ ViT Image Encoder (Patches) ] ──► [ Transformer Text Decoder ] ──► "WALMART"

1. Encoder: Pre-trained Vision Transformer (DeiT / BEiT) processes image patches.
2. Decoder: Pre-trained Text Transformer (RoBERTa / GPT-2) generates text tokens autoregressively.

TrOCR achieves state-of-the-art accuracy on printed text, handwritten documents, and historical manuscripts.

Say this out loud

OCR pipelines combine Stage 1 Text Detection (DBNet) to crop text polygons and Stage 2 Text Recognition (CRNN or TrOCR) to transcribe images into text strings. CTC loss trains text recognition models without character-level pixel segment annotations by introducing blank tokens and collapsing repeated predictions. Modern systems like TrOCR use Vision Transformer encoders and text decoders for end-to-end transcription.

Follow-ups to expect

• What is LayoutLM (Microsoft)? A multimodal Document Understanding Transformer that combines text embeddings, 2D spatial bounding box coordinates, and visual image features to parse structured PDF forms and invoices.
• How do OCR models handle oriented or curved text? Stage 1 detectors use Thin Plate Spline (TPS) transformation networks to rectify curved or tilted text strips into straight horizontal image rectangles before feeding them to Stage 2 recognition.