Design: Document Data Extraction

The Problem

Design a system that takes scanned or photographed documents (invoices, receipts, tax forms) and extracts structured data like vendor name, date, invoice number, line items, and total amount. The company processes 100,000 documents per day from thousands of different vendors, each with a different layout.

High-Level Architecture

  Scanned Document Image (PDF/JPEG)
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 1: PREPROCESSING         │
  │ Deskew, denoise, page detect  │
  │ Output: Clean document image  │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 2: OCR                    │  (~200ms)
  │ Extract text + bounding boxes │
  │ Output: Words with (x,y,w,h) │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 3: FIELD EXTRACTION       │  (~100ms)
  │ Layout-aware model (LayoutLM) │
  │ or template matching           │
  │ Output: Structured key-value  │
  │ pairs (date, total, vendor)   │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 4: VALIDATION & REVIEW   │
  │ Business rule checks          │
  │ Low-confidence → human review │
  └───────────────────────────────┘

Step 1: Document Preprocessing

Raw scanned documents are messy:

• Deskewing: Straighten rotated scans.
• Denoising: Remove scanner artifacts, shadows, and background noise.
• Page Detection: For multi-page PDFs, detect page boundaries and process each page separately.
• Document Classification: Determine the document type (invoice, receipt, W-2 tax form) to route to the appropriate extraction model.

Step 2: OCR (Optical Character Recognition)

Extract raw text and its spatial position on the page:

• Each detected word gets a bounding box: $(x, y, \text{width}, \text{height})$.
• Modern OCR engines (Google Cloud Vision, AWS Textract, Tesseract) achieve high accuracy on printed text.
• Handwritten text is significantly harder and may need specialized models.

The output is a list of words with their positions, like: {"text": "Invoice", "x": 50, "y": 20, "w": 80, "h": 15}.

Step 3: Field Extraction

This is the core ML challenge. Two approaches:

Approach A: Layout-Aware Transformer (LayoutLM)

LayoutLM (and its successors LayoutLMv2, LayoutLMv3) is a Transformer model that takes as input:

• Text token embeddings (same as BERT).
• 2D position embeddings (x and y coordinates of each word on the page).
• Image features (visual appearance of document regions).

Fine-tune LayoutLM on labeled document examples where each word is tagged with its field type (VENDOR_NAME, DATE, TOTAL_AMOUNT, LINE_ITEM, OTHER). This is essentially a token classification (NER) task augmented with spatial information.

Approach B: Template Matching for Known Vendors

For the 100 highest-volume vendors whose invoice layouts never change:

• Define templates specifying where each field appears spatially.
• Match incoming documents to templates using layout fingerprinting.
• Extract fields at known positions.

Use the LayoutLM approach for unknown or rare vendor formats.

Step 4: Validation and Human Review

Apply business rules to check extracted data:

• Does the total amount equal the sum of line items?
• Is the date in a valid format and range?
• Does the vendor name match a known vendor in the database?

Route low-confidence extractions to a human review queue. Human corrections feed back as training data.

Say this out loud

Document data extraction converts unstructured document images into structured fields. OCR extracts text with spatial coordinates. Layout-aware models like LayoutLM combine text embeddings with 2D position information to understand which text is the invoice number versus the date versus a line item. Template matching handles known formats while general models handle unknown layouts.

Followups to expect

1. How do you handle tables in documents? Use table detection models to identify table boundaries, then extract rows and columns using spatial clustering of OCR bounding boxes.
2. Can you use a multimodal LLM (GPT-4V) instead of LayoutLM? Yes, for simpler extraction tasks you can send the document image to a vision LLM with a structured extraction prompt. This is simpler to set up but more expensive per document and less controllable than a fine-tuned LayoutLM.