Feature Engineering Fundamentals
Better data beats fancy algorithms. Where 80% of real-world model accuracy improvements actually come from.
Encodings matched to model family
Feature engineering choice depends entirely on the model downstream:
| Technique | Linear Models / Neural Nets | Tree Models (GBDT / RF) |
|---|---|---|
| Feature Scaling | Mandatory (StandardScaler / MinMax) | Not required (Invariant to monotonic scaling) |
| Low-Cardinality Categoricals | One-Hot Encoding | Categorical / Ordinal / One-Hot |
| High-Cardinality Categoricals | Target Encoding / Entity Embeddings | Target Encoding (Out-of-Fold) / Frequency |
| Non-Linear Relationships | Log transform, Polynomials, Splines | Learned naturally via recursive splits |
| Feature Interactions | Explicit crosses (x1 × x2) | Learned implicitly along tree branches |
| Missing Values | Imputation (Mean/Median/Iterative) | Native handling (sent to best split direction) |
Handling continuous variables & non-linear transformations
- Log Transformation (
log1p(x)): Re-shapes right-skewed heavy-tailed distributions (income, price, transaction amount) into near-Gaussian shapes, reducing outlier distortion. - Box-Cox / Yeo-Johnson: Automated power transformations to stabilize variance and normalize distributions.
- Binning / Discretization: Converts continuous feature into categorical bins (quantile or uniform), enabling linear models to capture non-linear step functions.
Time & cyclical features
Timestamp columns hold rich signal when decomposed:
- Decomposition: Extract
is_weekend,is_holiday,day_of_week,quarter_end. - Cyclical Encoding: Map periodic time variables (hour 0-23, month 1-12) onto a 2D unit circle:
x_sin = np.sin(2 * np.pi * hour / 24.0)
x_cos = np.cos(2 * np.pi * hour / 24.0)
This guarantees Distance(23h, 0h) == Distance(0h, 1h).
Say this out loud
"Feature engineering must be tailored to model choice. Linear models and neural networks require strict normalization, one-hot encoding, log transforms for skewed inputs, and cyclical sin/cos encodings for periodic time features. Tree models are invariant to monotonic scaling, but benefit immensely from out-of-fold target encoding for high-cardinality categories, domain-specific interaction ratios, and window aggregations."
Follow-ups to expect
- What is target leakage in feature engineering? Computing statistics using information from the target variable that would not be available at inference time (e.g. computing global target encoding means across the entire dataset instead of strictly inside cross-validation training folds).
- How do you create interaction features for tabular data? Combine domain features as ratios (
Income / Household_Size,Debt / Revenue) or products (Price × Quantity). Tree models can learn these, but explicit ratios cut tree depth required. - How do you handle missing values in production? Track missingness as an explicit binary feature (
x_is_missing = 1), impute using training fold median for linear models, or let XGBoost learn default split directions natively during training.
Check yourself
Why should cyclical time features like 'hour of day' (0–23) be transformed into sine and cosine pairs rather than passed as raw integers to a Logistic Regression model?