Turning Text into Features

Converting Unstructured Text into Numbers

Machine learning algorithms cannot process raw text strings directly.

Text must be transformed into numeric features.

The approach used depends on whether you need fast sparse keyword representations or dense semantic representations.

Raw Text: "Machine learning is great"
               │
               ├─► Sparse Representation (TF-IDF / N-Grams)  ──► [0, 0, 1.4, 0, 0.8, ...]
               │
               └─► Dense Representation (Transformers/Embeddings) ──► [0.24, -0.81, 0.12, ...]

1. Traditional Sparse Representations

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. BAG OF WORDS (BOW)    │ 2. N-GRAMS               │ 3. TF-IDF                │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Counts word occurrences  │ Extracts adjacent word   │ Weights word frequency by│
│ in a fixed vocabulary    │ combinations ("not good")│ inverse document         │
│ dictionary. Ignores order│ to preserve local phrase │ frequency. Down weights  │
│ completely!              │ context.                 │ common stop words!       │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Term Frequency Inverse Document Frequency (TF-IDF)

$$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \log\left(\frac{|D|}{|{d \in D : t \in d}|}\right)$$

• Term Frequency (TF): How often word $t$ appears in document $d$.
• Inverse Document Frequency (IDF): Penalizes words that appear in almost all documents (such as "the", "and", "is").

2. Modern Dense Semantic Representations

Traditional sparse representations suffer from Vocabulary Mismatch:

• Sparse models see "doctor" and "physician" as two completely distinct, orthogonal vector dimensions.

Dense Vector Embeddings (from BERT or Sentence Transformers) map text into a continuous $d$-dimensional vector space ($d = 768$):

• Words with similar meanings sit near each other in vector space.
• Captures context, synonyms, and multi-word semantics.

Text Preprocessing Checklist

Before extracting text features:

1. Lowercasing: Standardize text to lowercase.
2. Punctuation Removal: Strip punctuation marks unless emoticons or code syntax matter.
3. Stop Word Removal: Remove uninformative words for sparse models.
4. Stemming / Lemmatization: Reduce words to base root forms ("running" $\to$ "run").

Say this out loud

Turning text into features converts raw strings into numeric representations. Sparse methods like Bag of Words, N-grams, and TF-IDF count word tokens and down weight common stop words. Dense embeddings from Transformer models map text into continuous semantic vector spaces, capturing synonyms and context.

Followups to expect

1. When would you choose TF-IDF over dense Transformer embeddings? TF-IDF is faster, cheaper, requires zero GPU infrastructure, and outperforms dense embeddings when matching exact part numbers, rare acronyms, or specific error codes.
2. What is Subword Tokenization (BPE / WordPiece)? Tokenization methods that break words into subword pieces (for example "unhelpfulness" $\to$ ["un", "help", "ful", "ness"]), preventing out of vocabulary errors on rare words.