Feature Selection Methods

Why Feature Selection Matters

Adding more features is not always better.

Including uninformative or redundant features causes serious issues:

• Overfitting (Curse of Dimensionality): The model learns noise in non-predictive columns.
• Slower Training and Inference: Processing 500 features takes far longer than 50 features.
• Increased Hosting Costs: Storing and computing unused features wastes data pipeline resources.

Feature Selection identifies the smallest subset of features that retains maximum predictive power.

Raw Feature Pool (1,000 Columns) ──► [ FEATURE SELECTION ] ──► Optimal Subset (50 Columns)

The 3 Feature Selection Families

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. FILTER METHODS        │ 2. WRAPPER METHODS       │ 3. EMBEDDED METHODS      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Fast statistical tests   │ Uses ML model to evaluate│ Model performs feature   │
│ (Correlation, Chi-Square,│ feature subsets          │ selection during training│
│ Variance Threshold).     │ (Recursive Feature       │ (L1 Lasso, Tree Feature  │
│ Independent of model!    │ Elimination). Heavy!     │ Importance). Best balance!│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Filter Methods (Fast & Model Independent)

Evaluate features individually based on statistical relationships with target labels:

• Variance Threshold: Remove zero-variance features (columns where values never change).
• Pearson Correlation: Remove features highly correlated with each other (multicollinearity).
• Chi-Square / Mutual Information: Measure statistical dependency between feature and target label.

2. Wrapper Methods (Search-Based)

Use a predictive model as an evaluation engine to test feature combinations:

• Recursive Feature Elimination (RFE): Train a model, rank feature importances, discard the weakest feature, and repeat until the desired subset size is reached.
• Forward Selection: Start with zero features and iteratively add the single feature that improves model validation score most.

3. Embedded Methods (Built Into Model Training)

Algorithms that perform feature selection natively during training:

• L1 Lasso Regularization: Penalizes absolute weight magnitudes, driving uninformative feature weights to exact zero.
• Tree-Based Feature Importance: XGBoost or Random Forests calculate feature split frequency or Gini impurity reductions.

Say this out loud

Feature selection removes redundant and uninformative features to reduce overfitting, speed up inference latency, and improve model interpretability. Filter methods use fast statistical tests like correlation or mutual information. Wrapper methods search feature subsets using model feedback. Embedded methods like L1 Lasso regularization drive uninformative weights to exact zero during training.

Followups to expect

1. What is Multicollinearity in feature selection? When two or more input features are highly correlated with each other, inflating weight variance in linear models and making feature importances unstable.
2. What is SHAP (Shapley Additive exPlanations) for feature selection? A game theoretic approach that measures the exact marginal contribution of each feature to model predictions, providing robust feature ranking.