Feature Crosses & Interactions

What is a Feature Cross?

A Feature Cross is a synthetic feature formed by combining (crossing) two or more categorical or discretized numerical variables:

$$\text{Feature Cross } X_1 \times X_2 = \text{AND}(X_1, X_2)$$

Feature 1 (Latitude Bucket):  "Lat_37"
Feature 2 (Longitude Bucket): "Long_122"
Feature Cross (Location):     "Lat_37_AND_Long_122"  (Unique Neighborhood Grid Cell!)

Why Feature Crosses Are Necessary

Linear models (like Logistic Regression) compute prediction logits as linear sums of independent weights:

$$y = w_1 X_1 + w_2 X_2 + b$$

A linear model can learn that Device = iPhone increases click probability, and that Country = US increases click probability.

However, a linear model cannot learn that the specific combination iPhone AND US has a super-additive non-linear effect unless you explicitly feed it the crossed feature iPhone_X_US!

┌──────────────────────────┬──────────────────────────┐
│ WITHOUT FEATURE CROSS    │ WITH FEATURE CROSS       │
├──────────────────────────┼──────────────────────────┤
│ Model treats features    │ Model assigns a dedicated│
│ independently:           │ weight to the specific   │
│ w_iphone + w_us          │ interaction:             │
│ (Fails non-linear rules!)│ w_iphone_x_us            │
└──────────────────────────┴──────────────────────────┘

Common Types of Feature Crosses

1. Categorical $\times$ Categorical: User_Gender $\times$ Product_Category (for example Female_X_Electronics).
2. Discretized Numerical $\times$ Categorical: Age_Bucket $\times$ Income_Bucket (for example Age25to34_X_HighIncome).
3. Spatial Crosses: Binned Latitude $\times$ Binned Longitude (creates bounding box locations for real estate models).

Automated Feature Crosses: DCNv2

In industrial applications with thousands of features, manually engineering all possible pairwise or triple feature crosses is impossible.

Architectures like Deep & Cross Network v2 (DCNv2) introduce explicit Cross Layers:

$$x_{l+1} = x_0 \odot (W_l x_l + b_l) + x_l$$

Cross layers automatically calculate degree 1, degree 2, and degree 3 feature interactions explicitly without manual feature engineering, sharing representation layers with deep MLP stacks.

Say this out loud

Feature Crosses combine two or more categorical variables to form composite features. They allow linear models to learn non linear feature interaction rules without manual non linear transformations. Modern architectures like Deep and Cross Networks compute degree 1, 2, and 3 feature interactions automatically across high cardinality data.

Followups to expect

1. What is the impact of feature crosses on model sparsity and RAM? Crossing high cardinality categorical features creates an exponential explosion of unique sparse string keys, requiring L1 regularization or hash trick hashing to keep memory manageable.
2. Can decision trees learn feature interactions natively? Yes, decision trees naturally learn non-linear feature interactions by splitting sequentially on different features along decision paths.