Data Leakage

The three families

1. Target leakage. A feature encodes the outcome. account_closed_date in a churn model. num_collection_calls in a default model. days_in_ICU for a mortality model. These are consequences of the label, generated after the decision point.

Detection: if a single feature carries suspiciously high importance, interrogate it. Ask the data owner when the column is populated in the source system. If the answer involves anything downstream of the event, drop it.

2. Train–test contamination. Information crosses the split:

• Scaler/imputer/encoder fitted on the whole dataset before splitting.
• Feature selection run on all rows before splitting.
• Duplicate or near-duplicate rows in both sides (very common in scraped text and image datasets).
• The same user/patient/document split across train and test — you measure entity memorisation.
• Oversampling before splitting.

3. Temporal leakage. The model sees the future:

• Shuffled k-fold on time series.
• Aggregates computed over the full history rather than as-of the prediction time (user_lifetime_purchase_count including purchases after the label date).
• Joining a dimension table whose values have been overwritten since — you get today's customer_segment, not the segment as of the event.

Point-in-time correctness

The industrial answer, and the reason feature stores exist. Every feature must be computed from data whose timestamp is strictly before the prediction timestamp. A backfill that joins on entity ID alone, ignoring event time, is the single most common source of silent leakage in production ML — and the direct cause of training–serving skew, where offline metrics look great and online performance does not match.

Practical guard: build features via an as-of join keyed on (entity_id, event_time), and add a gap equal to your prediction horizon.

Your checklist

• Every transform lives inside a Pipeline fitted per fold.
• Split first, always. Resample, select, encode after.
• Grouped data → GroupKFold by the entity you must generalise across.
• Time series → forward-chaining splits with a horizon gap.
• Deduplicate before splitting (hash rows / near-duplicate detection).
• For each feature, answer in writing: at prediction time, would this value exist, and would it have this value?
• Audit the top-5 features by importance for plausibility.
• Sanity check: if offline metrics beat the human expert by a wide margin, assume leakage until proven otherwise.

Say this out loud

"Anything that looks too good is leakage until proven otherwise. I'd check three things: whether any feature is a consequence of the label rather than a predictor, whether preprocessing or duplicate rows crossed the split, and whether features are point-in-time correct relative to the prediction timestamp. The structural fixes are per-fold pipelines, grouped or time-based splits with a horizon gap, and as-of joins on event time."