Using Embeddings as Features

What Are Embedding Features?

High-dimensional unstructured data (text documents, images, audio clips, user interaction graphs) cannot easily be fed into traditional tabular models like XGBoost.

Instead of writing manual rules or calculating basic statistics, we pass unstructured data through a Pre-Trained Deep Learning Backbone to extract a continuous Dense Vector Embedding ($d = 128 \text{ to } 768$ dimensions).

Unstructured Input (Text / Image / Graph Entity)
                       │
                       ▼
  [ PRE-TRAINED BACKBONE MODEL (BERT / ResNet / Node2Vec) ]
                       │
                       ▼
  Dense Feature Vector e = [0.14, -0.82, 0.45, ..., 0.08]
                       │
                       ▼
  Passed as Input Features to Downstream Model (XGBoost / Logistic Reg)

3 Core Sources of Embedding Features

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. TEXT EMBEDDINGS       │ 2. VISION EMBEDDINGS     │ 3. GRAPH EMBEDDINGS      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Sentence Transformers /  │ ResNet / Vision          │ Node2Vec / GraphSAGE     │
│ OpenAI Embeddings.       │ Transformer activations. │ graph node embeddings.   │
│ Captures document topic  │ Captures visual style    │ Captures social graph    │
│ and semantic intent!     │ and object features!     │ network connections!     │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Pooling Sequences into Fixed Features

When an entity consists of multiple embeddings (for example a user who clicked 5 product items), how do we represent the sequence as a single feature vector?

Common Pooling Strategies:

1. Mean Pooling: Take the element-wise average across item vectors:

$$\mathbf{e}{\text{user}} = \frac{1}{K} \sum{i=1}^K \mathbf{e}_{\text{item}_i}$$

2. Max Pooling: Take the element-wise maximum across dimensions to capture peak interest signals.
3. Attention Weighted Pooling: Weight recent item vectors higher than older item vectors using self-attention.

Frozen vs Fine-Tuned Embeddings

• Frozen Embeddings: Extract static embedding vectors once and store them in a Feature Store. Downstream models treat vectors as fixed input features. Fast, cheap, and prevents overfitting on small datasets.
• Fine-Tuned Embeddings: Backpropagate downstream task loss back through the embedding generator layers during training. Provides higher task accuracy when abundant training data is available.

Say this out loud

Using embeddings as features extracts continuous dense vectors from pre trained deep learning backbones to represent text, images, or graph entities in downstream models. Sequence inputs are combined using mean or attention pooling. Frozen embeddings offer fast, cheap feature generation, while fine tuning updates embedding representations for maximum downstream accuracy.

Followups to expect

1. How do you handle dimensionality reduction on 768-dim embeddings for GBDT models? Use Principal Component Analysis (PCA) or UMAP to compress 768-dimensional embeddings down to 32 or 64 dimensions before feeding them into Gradient Boosted Trees.
2. What is Item2Vec? Applying Word2Vec skip-gram algorithms to user interaction sessions, treating item click sequences like sentences to learn item embedding representations.