Feature Engineering Patterns — A Field Guide

Feature engineering is where most of the accuracy gain in a tabular ML project actually comes from — not model choice. Swapping XGBoost for a neural net on the same raw columns rarely moves the needle as much as one well-constructed interaction feature or a correctly aggregated time window. This is a working set of patterns, not a survey of every technique.

Aggregations are the highest-leverage feature type

For any entity with a history — a customer, a device, an account — rolling aggregations (count, sum, mean, min/max over a trailing window) tend to carry more signal than the raw event stream. "Number of failed logins in the last 24 hours" beats feeding a model the raw login timestamps and letting it infer the pattern. The key discipline is picking window lengths that map to something real about the domain (a billing cycle, a session, a business week) rather than an arbitrary round number.

Avoid target leakage in time-based features

The single most common feature engineering bug is computing an aggregate using data that wouldn't have been available at prediction time. A "customer lifetime value" feature computed over the customer's entire history, then used to predict an event from the middle of that history, leaks the future into the past. Every time-based feature needs an explicit as-of timestamp, and the aggregation window must end at or before that timestamp — this is exactly the point-in-time correctness problem that tools like Feast exist to solve at scale.

Categorical encoding choices

One-hot encoding works fine for low-cardinality categoricals but blows up dimensionality for anything with hundreds of distinct values (zip codes, SKUs, merchant IDs). Target encoding (replacing a category with the mean of the target for that category) is more compact but needs to be fit only on training folds — computing it on the full dataset before a train/test split is another common leakage source. Tree-based models (gradient boosting, random forests) also handle raw categorical codes or frequency encoding reasonably well without one-hot expansion.

from sklearn.model_selection import KFold
import numpy as np

def kfold_target_encode(df, col, target, n_splits=5):
    encoded = np.zeros(len(df))
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    for train_idx, val_idx in kf.split(df):
        means = df.iloc[train_idx].groupby(col)[target].mean()
        encoded[val_idx] = df.iloc[val_idx][col].map(means)
    return encoded

Feature scaling matters less than people think — for some models

Gradient boosting and random forests are invariant to monotonic transformations of individual features, so scaling and normalization buy nothing there. Linear models, SVMs, and neural networks are scale-sensitive and need standardization or normalization or training will be slower and less stable. Knowing which family you're feeding the features to saves time spent scaling features that didn't need it.

Feature selection as a discipline, not an afterthought

More features are not free — each one is a maintenance liability, a potential source of drift, and a slower inference path. Permutation importance and SHAP values are the two most reliable tools for identifying which engineered features actually contribute versus which are just adding noise the model happened to fit. Pruning a feature set down after the fact is normal and often improves generalization, not just speed.

Wrapping up

The patterns that consistently pay off are domain-grounded aggregation windows, leakage-safe encoding fit per training fold, and pruning features based on measured importance rather than intuition. Model architecture is usually the smaller lever — the feature set is where the real accuracy gains live.