Wide & Deep, DeepFM

The Recommendation Engine Dilemma

Industrial recommendation systems (Google Play Store, YouTube, E-commerce) face two competing goals:

1. Memorization: Learning historical user-item co-occurrence rules ("Users who installed app Netflix also installed app Hulu").
2. Generalization: Recommending fresh, unseen items based on semantic feature similarity ("User likes Sci-Fi games, so recommend new Space RPG").

  Traditional Linear Models (Logistic Regression):   Great at MEMORIZATION, terrible at Generalization.
  Standard Deep Neural Networks (MLP):             Great at GENERALIZATION, bad at Memorizing specific rules.

Wide & Deep Learning (Cheng et al., 2016 / Google) unifies both paradigms into a single hybrid network.

                  WIDE & DEEP MODEL ARCHITECTURE

  Output:  y_pred = Sigmoid( W_wide^T [x, ϕ(x)]  +  W_deep^T a_L  +  b )
                                ▲                     ▲
                                │                     │
            ┌───────────────────┴─────────────────────┴───────────────────┐
            │                                                             │
     [ WIDE COMPONENT ]                                          [ DEEP COMPONENT ]
   Generalized Linear Model                                     Deep MLP Network
   (Memorizes Cross-Products)                                   (Generates Embeddings)
            │                                                             │
  Sparse Categorical Features                                   Dense Embedding Lookup
  (e.g. AND(User_Impression, Item_App))                         (e.g. User & Item Embeddings)

1. The Wide Component (Memorization)

The Wide Component is a Generalized Linear Model:

$$y_{\text{wide}} = w^T x + b$$

Input features $x$ include raw categorical features and Cross-Product Transformations $\phi_k(x)$:

$$\phi_k(x) = \prod_{j=1}^d x_j^{c_{kj}}, \quad c_{kj} \in {0, 1}$$

• Example Cross-Product: AND(User_Language == "English", Installed_App == "Netflix")
• If this boolean condition holds true $\implies \phi_k(x) = 1$.
• Strength: Easily memorizes high-value specific user-item combinations.

2. The Deep Component (Generalization)

The Deep Component is a Feed-Forward Neural Network:

Sparse categorical features (User ID, Item ID, Country) are mapped to Dense $d$-dimensional Embeddings:

$$a^{(0)} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_m]$$

Dense vectors pass through 3 to 4 fully connected ReLU layers:

$$a^{(l+1)} = \text{ReLU}\left(W^{(l)} a^{(l)} + b^{(l)}\right)$$

• Strength: Generalizes to novel user-item pairs that never co-occurred in historical training logs.

DeepFM: Automated Feature Interactions (Guo et al., 2017)

A major drawback of standard Wide & Deep is that the Wide Component requires manual feature engineering of cross-product transformations $\phi(x)$.

DeepFM replaces the Wide linear component with a Factorization Machine (FM):

  DeepFM = Factorization Machine (FM) Component  +  Deep MLP Component

The FM component calculates $2\text{nd}$-order pairwise feature dot products $\langle v_i, v_j \rangle$ automatically without manual feature engineering, sharing the exact same embedding vectors $v$ with the Deep component!

Say this out loud

Google's Wide and Deep architecture combines a Wide linear model for memorizing specific cross product feature rules with a Deep MLP for generalizing via dense embeddings. DeepFM replaces manual cross product engineering by using a Factorization Machine component to learn 2nd order feature interactions automatically.

Followups to expect

1. Why does Wide and Deep use Follow-The-Regularized-Leader (FTRL) for the Wide component? FTRL-Proximal is an optimization algorithm that enforces L1 sparsity on Wide linear weights, driving uninformative cross-product feature weights to exact zero.
2. What is DCN (Deep & Cross Network)? An extension that replaces the Wide component with explicit Cross Layers that compute degree-1, 2, 3... feature interactions automatically without manual engineering.