Stemming, Lemmatization & Stopwords

Traditional Text Preprocessing Pipeline

Before modern Transformers, raw text contained too many unique word variations for simple models to handle.

Traditional NLP pipelines cleaned text through a sequence of steps:

  Raw Text: "The 3 fast cats were RUNNING quickly after mice!"
       │
       ▼ Lowercasing & Punctuation Removal
  "the 3 fast cats were running quickly after mice"
       │
       ▼ Stopword Filtering (Remove "the", "were", "after")
  "3 fast cats running quickly mice"
       │
       ▼ Lemmatization / Stemming
  "3 fast cat run quick mouse"

1. Stopword Removal

Stopwords are high frequency words that appear across almost all documents:

• Examples: "the", "is", "at", "which", "on", "and".

In classical TF-IDF models, removing stopwords reduced vocabulary noise and saved memory.

Why Modern Models Avoid Stopword Removal

Removing stopwords can completely destroy sentence meaning:

  Original:  "Flight from Paris to London is NOT delayed."
  No Stopwords: "Flight Paris London delayed"  <-- Opposite meaning! Negation lost!

Transformers require raw un-filtered text to process grammatical relationships and self attention context correctly.

2. Stemming vs Lemmatization

Both techniques reduce inflected words to their base root form.

┌──────────────────────────┬──────────────────────────┐
│ STEMMING (Porter / Snowball)| LEMMATIZATION (WordNet)│
├──────────────────────────┼──────────────────────────┤
│ Uses crude rule-based    │ Uses full dictionary     │
│ suffix chopping.         │ morphological lookup.    │
│ Fast, but produces invalid│ Slower, but returns valid│
│ words ("running" -> "runni").| base words ("better" -> "good").|
└──────────────────────────┴──────────────────────────┘

Stemming (Porter / Snowball Stemmer)

Applies simple heuristic rules to chop off common prefixes and suffixes:

• "running" $\to$ "runni"
• "studies" $\to$ "studi"
• "cats" $\to$ "cat"

Fast, but crude. Words often get chopped into non-dictionary strings.

Lemmatization (WordNet Lemmatizer)

Uses vocabulary dictionaries and part of speech (POS) grammar tags to map words to their true morphological base form (Lemma):

• "running" (Verb) $\to$ "run"
• "better" (Adjective) $\to$ "good"
• "mice" (Noun) $\to$ "mouse"

Accurate, but slower. Requires morphological dictionary lookups.

The Shift to Subword Tokenization

Modern LLMs (GPT-4, LLaMA 3) do NOT use lowercasing, stopword removal, stemming, or lemmatization!

Instead, Byte Pair Encoding (BPE) handles morphological variations automatically by breaking rare words into subwords ("un-friend-li-ness"), preserving raw text fidelity.

Say this out loud

Stemming chops off word endings using crude rules, often producing non-dictionary words. Lemmatization uses vocabulary dictionaries to return valid base root words. Traditional NLP pipelines used lowercasing, stopword removal, and lemmatization to reduce vocabulary noise, but modern Transformers operate directly on raw text using Byte Pair Encoding subwords.

Followups to expect

1. What is Part of Speech (POS) Tagging? Labeling each word in a sentence with its grammatical role (Noun, Verb, Adjective, Adverb) based on sentence context.
2. When is lowercasing harmful in NLP? In Named Entity Recognition (NER), where capitalization is a primary signal for identifying proper nouns (for example, "White House" vs "white house").