Feature Engineering and Preprocessing

Feature engineering and preprocessing turn raw columns into the numerical representation a model can actually learn from. In this lesson, the outcome is practical: choose transformations for numeric, categorical, skewed, and missing values; fit those transformations only on training data; and package them so the exact same learned rules are applied during validation and prediction.

This belongs in the Improving Models section because many apparent model improvements are really representation improvements. A linear model cannot use the string pro directly, a distance-based model is distorted when income is measured in thousands and age in decades, and a tree can overfit a noisy identifier. Good preprocessing makes the learning problem closer to the real signal without letting information from the future or validation fold leak into training.

Purpose and Outcome

Raw data usually mixes measurement types. Numeric columns may need imputation, scaling, clipping, binning, or nonlinear transforms. Categorical columns need an encoding such as one-hot, ordinal, target, hashing, or learned embeddings. Dates often become durations, cyclical components, or event flags. Text and images need specialized vectorizers. The purpose is not to decorate a dataset with more columns; it is to expose stable predictive structure while preserving the rule that every feature must be available at prediction time.

After this chapter you should be able to inspect a dataset, identify which transformations are fitted and which are stateless, build a preprocessing pipeline, reason about trade-offs, diagnose common failures, and verify that preprocessing is evaluated honestly with the model.

How Preprocessing Works Internally

A preprocessing object usually has two phases. During fit, it learns state from training data: a scaler learns means and standard deviations, an imputer learns medians or modes, a one-hot encoder learns the set and order of categories, and a text vectorizer learns vocabulary. During transform, it applies that stored state to new rows. The separation matters because learning state from the full dataset before splitting silently contaminates validation. The validation data has influenced the representation, even if the final estimator has not seen its target labels.

A pipeline enforces that order. In scikit-learn terminology, a Pipeline chains steps, where intermediate steps implement fit and transform, and the final step is usually an estimator with fit and predict. A ColumnTransformer routes different columns through different transformations and concatenates their outputs into a feature matrix. Its output column order is determined by the order of transformers and by the learned names inside each transformer. That matrix becomes the model input.

Three internal details drive most design decisions. First, transformed features must have fixed meaning across train and inference. If category index 3 means region=west during training and region=east during serving, predictions are invalid. Second, many transformations change scale or distribution but not information content; they help some estimators more than others. Third, transformations can increase dimensionality. One high-cardinality categorical feature may become thousands of sparse columns, affecting memory, latency, and regularization.

API Anatomy

A robust preprocessing specification names columns, chooses per-column transformers, and keeps the fitted object with the model artifact. This skeleton shows the shape without fitting any data:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_features = ["age", "income"]
categorical_features = ["plan", "region"]

numeric_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

categorical_steps = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

preprocess = ColumnTransformer([
    ("num", numeric_steps, numeric_features),
    ("cat", categorical_steps, categorical_features),
])

The numeric path first fills missing values with training medians, then standardizes each numeric column to approximately zero mean and unit variance using training statistics. The categorical path fills missing values with the most common training category, then creates indicator columns. handle_unknown="ignore" means an unseen category at prediction time produces all zeros for that feature group instead of raising an exception. That behavior avoids serving failures, but it can hide drift, so you should still monitor unknown-category rates.

Example 1: Encode and Scale a Small Table

The first worked example fits a scaler and one-hot encoder on a tiny training table. It demonstrates learned state and output feature names:

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame({
    "age": [20, 30, 40],
    "plan": ["free", "pro", "free"],
})

preprocess = ColumnTransformer([
    ("age", StandardScaler(), ["age"]),
    ("plan", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["plan"]),
])

matrix = preprocess.fit_transform(train)
print(matrix.round(2).tolist())
print(preprocess.get_feature_names_out().tolist())

The deterministic output is approximately [[-1.22, 1.0, 0.0], [0.0, 0.0, 1.0], [1.22, 1.0, 0.0]], followed by feature names similar to ['age__age', 'plan__plan_free', 'plan__plan_pro']. The age values are centered and scaled. The category free becomes one indicator column and pro becomes another. A future row with plan="enterprise" would receive zeros for both known plan columns because the encoder was told to ignore unknown categories.

Example 2: Transform a Skewed Numeric Feature

Many business features such as purchase amount, session duration, and account balance are right-skewed: most values are small, while a few are very large. A log transform compresses large values and often makes linear relationships easier for linear models to learn:

import numpy as np
import pandas as pd
from sklearn.preprocessing import FunctionTransformer

amounts = pd.DataFrame({"amount": [0.0, 9.0, 99.0, 999.0]})
log_transform = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
result = log_transform.fit_transform(amounts)
print(result.round(3).ravel().tolist())

The output is [0.0, 2.303, 4.605, 6.908]. log1p computes log(1 + x), so zero stays zero and positive values are compressed. This is useful when each multiplication in the raw feature corresponds to an additive change in risk or demand. It is not valid for negative values without a deliberate shift or a different transform, and it can make explanations less direct because model coefficients now refer to log-scaled units.

Example 3: Evaluate the Whole Pipeline

The third example combines preprocessing and a classifier inside cross-validation. Each fold refits preprocessing only on that fold’s training portion, then scores on its held-out portion:

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X = pd.DataFrame({
    "hours": [1.0, 2.0, 4.0, 8.0, 16.0, 32.0],
    "source": ["ad", "search", "ad", "email", "search", "email"],
})
y = [0, 0, 0, 1, 1, 1]

model = Pipeline([
    ("prep", ColumnTransformer([
        ("num", StandardScaler(), ["hours"]),
        ("cat", OneHotEncoder(handle_unknown="ignore"), ["source"]),
    ])),
    ("clf", LogisticRegression()),
])

scores = cross_val_score(model, X, y, cv=3, scoring="accuracy")
print(scores.tolist())

The exact accuracy list is deterministic for a fixed library implementation, but the important behavior is procedural: the scaler, encoder, and logistic regression are fitted fresh inside each fold. This is the correct pattern. If you call fit_transform on all of X before cross_val_score, the held-out fold helps define means, variances, and categories. That leakage can make validation scores look better than real deployment performance.

Design Choices and Trade-offs

Scaling is essential for distance-based models, gradient descent, principal components, and regularized linear models because feature magnitude affects the optimization path and penalty. It is usually less important for tree ensembles because splits depend on order rather than absolute scale. Standardization preserves outliers; robust scaling uses medians and quantiles to reduce outlier influence; min-max scaling is easy to interpret but fragile when future values exceed the training range.

Categorical encoding has sharper trade-offs. One-hot encoding is simple, preserves no false ordering, and works well for low-cardinality features. It can explode feature count for user IDs, product SKUs, or postal codes. Ordinal encoding is compact but implies an order; it is appropriate only when the model can handle arbitrary integer labels or when the categories are truly ordered. Target encoding can be powerful for high-cardinality categories, but it must be computed inside cross-validation or with smoothing because it can leak target information. Hashing fixes dimensionality and handles new categories naturally, but collisions make interpretation harder.

Feature construction should follow the prediction moment. A churn model may use days_since_last_login calculated as of the scoring date, not using any activity after that date. Aggregations such as rolling counts must have explicit windows. Missingness may itself be informative, so sometimes you add a missing indicator in addition to imputation. Dropping rows is simple but can bias training if missingness is systematic.

Failure Modes and Troubleshooting

Symptom: validation accuracy is excellent, but production accuracy drops quickly. Cause: preprocessing was fitted before the train-test split, or an aggregate feature used future data. Diagnostic steps: trace where fit is called, inspect feature timestamps, and run cross-validation with a pipeline that owns all fitted transformations. Correction: move every fitted preprocessing step inside the pipeline and use time-aware splits when the prediction is chronological.

Symptom: prediction fails with an error about unknown categories. Cause: the encoder learned categories during training and encountered a new value during inference. Diagnostic steps: log the column and category value without sensitive payloads, compare serving categories with training categories, and check whether a data contract changed. Correction: use an unknown-category strategy, retrain when new categories are meaningful, and monitor the unknown rate as drift.

Symptom: training or inference becomes slow and memory-heavy after adding categorical features. Cause: one-hot encoding expanded high-cardinality columns into a very wide matrix. Diagnostic steps: print transformed matrix shape, count categories per column, and inspect sparse versus dense output. Correction: group rare categories, use hashing or target encoding with leakage controls, or choose a model that handles categorical features more directly.

Symptom: model explanations disagree with domain intuition. Cause: coefficients or feature importances are being read in transformed units, after scaling, logging, binning, or one-hot expansion. Diagnostic steps: retrieve transformed feature names, map each engineered feature back to source columns, and test monotonic changes on representative rows. Correction: document transformations with the model card and prefer explanation methods that operate on the same representation the model sees.

Security, Performance, and Reliability

Preprocessing can expose sensitive information. Rare categories may encode individual identities, and target encoding can memorize outcomes for small groups. Limit direct identifiers, apply minimum-frequency thresholds, and avoid logging raw rows. When preprocessing is part of a serving path, serialize the fitted transformer and estimator together so category order, imputation values, and scaling statistics cannot drift independently. Treat the artifact as a versioned unit.

Performance depends on matrix size, data type, and sparsity. Dense one-hot matrices waste memory when most entries are zero. Sparse matrices save memory but not every estimator accepts them efficiently. Reliability depends on schema checks: columns should be present, types should be coercible by design, and units should be stable. A silent change from dollars to cents can be more damaging than a hard failure.

Hands-on Lab

Prerequisites: Python with pandas and scikit-learn installed, plus a shell where you can run a single script. Use a small local dataset first so you can see every transformation.

1. Create a dataframe with two numeric columns, one categorical column, and a binary target. Include at least one missing numeric value and one rare category.
2. Split the dataframe into train and test before fitting any transformer.
3. Build a ColumnTransformer with median imputation and scaling for numeric columns and one-hot encoding for the categorical column.
4. Wrap the transformer and a classifier in a Pipeline.
5. Fit the pipeline on training data only, then call predict and score on the test data.
6. Print pipeline.named_steps["prep"].get_feature_names_out() to verify the transformed feature names.
7. Create one new scoring row containing an unseen category and verify that prediction still returns a class when unknown categories are ignored.

Verification: the transformed feature names should include scaled numeric columns and one indicator per category learned from training. The test score should be computed by the full pipeline, not by a model trained on preprocessed data from the entire dataset. The unseen category row should not crash if the encoder is configured for unknown values.

Cleanup: remove any temporary script, notebook output, and serialized model artifact you created. If you saved a pipeline file for experimentation, delete it or label it clearly as a toy artifact so it is not confused with a production model.

Assessment Exercises

• You have a feature merchant_id with 200,000 values. Compare one-hot encoding, hashing, and target encoding for a fraud model. Which risks would you test before choosing?
• A teammate scales all numeric columns before running train-test split because scaling does not use the target. Explain why this can still bias evaluation.
• Design a preprocessing plan for a house-price model with square footage, ZIP code, sale date, and missing renovation year. Which transformations are fitted?
• A model’s predictions changed after deployment even though the estimator file did not. List two preprocessing artifact mismatches that could cause this.
• Write a test that proves a pipeline can score a row containing a category that did not appear during training.

Summary

Feature engineering and preprocessing define the representation a model learns from. The key discipline is to distinguish raw data, fitted transformation state, transformed feature matrices, and estimator parameters. Fit preprocessing only on training data, keep it inside the evaluation and serving pipeline, choose encodings and scalers that match the model and data, and monitor schema, drift, and dimensionality. Better features improve models when they expose real predictive structure without leaking unavailable information.