Topic Modelling & LDA

What is Topic Modelling?

Imagine receiving 100,000 unlabelled customer support emails.

Reading them manually is impossible. Topic Modelling automatically discovers the hidden thematic topics in the dataset without human labels:

  UNLABELLED DOCUMENTS ──► [ TOPIC MODEL ] ──► DISCOVERED THEMATIC TOPICS:
                                                - Topic 1 (Billing):  "card", "charge", "refund", "invoice"
                                                - Topic 2 (Shipping): "delivery", "delay", "tracking", "fedex"
                                                - Topic 3 (Account):  "password", "login", "reset", "email"

1. Latent Dirichlet Allocation (LDA - Blei et al., 2003)

LDA is a classic generative probabilistic model.

The LDA Generative Story

LDA assumes every document is created through a 2-step random sampling process:

1. Pick a Topic Mixture for the document (e.g. 70% Politics, 30% Economics).
2. For each word position, pick a topic from the mixture, then sample a Word from that topic's word distribution.

  Document Mixture (Dirichlet α) ──► Topic Choice ──► Word Distribution (Dirichlet β) ──► Final Word

LDA uses Variational Inference or Gibbs Sampling to reverse this process, learning the hidden topic distributions from raw text.

Limitations of LDA

• Bag of Words: Ignores word order and context ("not good" is treated as separate words "not" and "good").
• Requires Pre-specifying K: You must manually choose the number of topics $K$ upfront.

2. BERTopic (Modern Neural Approach)

BERTopic (Grootendorst, 2022) upgrades topic modelling using modern Transformer embeddings:

  Raw Text ──► [ Sentence-BERT Embeddings ] ──► [ UMAP Dim Reduction ] ──► [ HDBSCAN Clustering ] ──► [ c-TF-IDF Topic Keywords ]

1. Sentence Embeddings: Extracts contextual vectors using Sentence-BERT, preserving semantic word order.
2. UMAP: Reduces high-dimensional embeddings to 5D space while preserving local neighbor clusters.
3. HDBSCAN: Automatically finds dense clusters without requiring you to guess $K$ upfront.
4. c-TF-IDF: Extracts the most representative keywords for each discovered cluster.

Evaluating Topic Models: Topic Coherence

How do you know if a topic model is good?

Topic Coherence ($C_v$): Measures how frequently a topic's top 10 keywords co-occur together in a reference text corpus. High coherence means the keywords form a clear, logical concept.

Say this out loud

Topic Modelling discovers hidden themes in unlabelled text collections. LDA assumes documents are mixtures of topics, and topics are mixtures of words. Modern BERTopic replaces LDA's bag-of-words approach by combining contextual Transformer embeddings with UMAP dimension reduction and HDBSCAN clustering to discover semantically coherent topics automatically.

Followups to expect

1. What is PyLDAVis? An interactive visualization tool that plots discovered LDA topics as 2D circles using Multidimensional Scaling (MDS), allowing users to inspect topic overlap and top keywords.
2. Can topic models be used for document classification? Yes. The topic probabilities $P(\text{Topic}_k \mid \text{Doc}_i)$ act as low-dimensional dense features for training downstream classifiers.