Module 5 · Improving Models

Overfitting and Regularization

Lesson 13 of 24 13 min read Updated

Overfitting happens when a model learns accidental detail from the training set so well that it stops representing the broader pattern you need at prediction time. Regularization is the family of techniques that deliberately limits that learning: penalties on coefficients, simpler features, early stopping, dropout, pruning, more data, or validation-driven model selection. The outcome for this lesson is practical: you should be able to diagnose high variance, choose an appropriate constraint, and verify that the change improves generalization rather than only making the training score look different.

In the Improving Models section, this topic is the counterweight to simply trying larger models. A model can be expressive enough to memorize unusual examples, label noise, duplicate records, or future information hidden in a feature. Regularization gives you a disciplined way to spend model capacity only where it buys repeatable predictive value.

Purpose and Learning Outcome

The plain-language test is simple: a useful model should perform well on new examples drawn from the same problem, not merely on the rows used to fit it. If training error is much lower than validation error, the model has high variance. It is reacting too strongly to the specific sample it saw. Regularization reduces variance by adding a preference for simpler explanations, even when a more complex explanation fits the training data more closely.

That preference is not a moral rule. Sometimes complex structure is real, and excessive regularization causes underfitting. The working skill is to compare training behavior, validation behavior, and domain expectations before choosing between collecting better data, simplifying inputs, strengthening a penalty, or changing the model class.

How It Works Internally

Most supervised training minimizes an objective function. For ordinary least squares regression, the objective is the sum of squared residuals between predictions and targets. Ridge regression changes the objective by adding an L2 penalty: the sum of squared coefficients multiplied by a strength parameter usually called alpha or lambda. Lasso adds an L1 penalty: the sum of absolute coefficient values. Logistic regression often exposes the inverse regularization strength as C, so smaller C means stronger regularization.

This changes the shape of the optimization problem. Without a penalty, many coefficient combinations may fit noise, especially when features are correlated or the number of features is large relative to the number of rows. L2 regularization shrinks coefficients smoothly toward zero, which tends to keep all features but makes each one less extreme. L1 regularization can drive some coefficients exactly to zero, which performs a kind of feature selection. Elastic net combines both penalties so correlated features can be handled more gently than pure lasso while still allowing sparsity.

The bias-variance trade-off is the central vocabulary. A highly flexible model has low bias because it can represent many functions, but it may have high variance because small changes in the training data can produce large changes in the fitted function. Regularization increases bias on purpose to reduce variance. The best setting is rarely the one with the lowest training error; it is the one with the lowest honest validation error under the metric that matches the task.

API and Configuration Anatomy

Regularization usually appears in one of four places. In linear models, it is a penalty parameter such as alpha, C, penalty, or l1_ratio. In tree ensembles, it appears as depth limits, minimum samples per split, minimum leaf size, learning rate, or subsampling. In neural networks, it appears as weight decay, dropout probability, early stopping patience, data augmentation, or smaller architecture choices. In feature engineering, it appears as lower polynomial degree, fewer text n-grams, stronger frequency cutoffs, or removal of leaky columns.

The split strategy is part of the anatomy. You need training data for fitting, validation data for choosing complexity, and test data for a final estimate after choices are frozen. Cross-validation repeats the validation idea across folds so the estimate is less dependent on one lucky split. Pipelines matter because scaling, imputation, and feature construction must be fit inside each training fold. Fitting preprocessing on the whole dataset before splitting leaks information and can make an overfit model look better than it is.

Example 1: Polynomial Complexity

This example compares a straight line with a fifth-degree polynomial on a small regression problem. The higher-degree model can bend through more training quirks. On tiny data, that extra flexibility often lowers training error while worsening test error.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

x_train = np.array([0, 1, 2, 3, 4, 5], dtype=float).reshape(-1, 1)
y_train = np.array([0.2, 0.9, 2.2, 2.8, 4.1, 5.0])
x_test = np.array([0.5, 1.5, 2.5, 3.5, 4.5], dtype=float).reshape(-1, 1)
y_test = np.array([0.4, 1.6, 2.4, 3.6, 4.4])

for degree in [1, 5]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(x_train, y_train)
    train_rmse = mean_squared_error(y_train, model.predict(x_train), squared=False)
    test_rmse = mean_squared_error(y_test, model.predict(x_test), squared=False)
    print(f"degree={degree} train_rmse={train_rmse:.3f} test_rmse={test_rmse:.3f}")

The exact numbers depend on the installed numerical libraries, but the expected pattern is that degree 5 has very low training RMSE and can have worse test RMSE than degree 1. The important interpretation is not that degree 5 is always bad. It is that model capacity must be evaluated away from the rows used to fit it.

Example 2: Ridge Shrinkage

Ridge regularization keeps the polynomial feature space but discourages large coefficients. The prediction at 2.5 shows how the fitted curve becomes less reactive as alpha increases.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline

x = np.array([0, 1, 2, 3, 4, 5], dtype=float).reshape(-1, 1)
y = np.array([0.2, 0.9, 2.2, 2.8, 4.1, 5.0])
query = np.array([[2.5]])

for alpha in [0.0, 0.1, 10.0]:
    model = make_pipeline(
        PolynomialFeatures(5, include_bias=False),
        StandardScaler(),
        Ridge(alpha=alpha)
    )
    model.fit(x, y)
    print(f"alpha={alpha:>4} prediction={model.predict(query)[0]:.3f}")

With alpha=0.0, ridge behaves like an unpenalized linear regression over the expanded features. Moderate alpha usually keeps the useful curve while reducing instability. Very large alpha can flatten the model too much, giving higher bias. This is why alpha is tuned on validation data instead of chosen by taste.

Example 3: Logistic Regularization Strength

Classification models overfit too. Logistic regression learns a boundary by optimizing log loss, and its regularization controls how steep or complex that boundary can become in feature space.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = np.array([
    [0.2, 1.1], [0.4, 0.9], [0.5, 1.3], [1.0, 0.7],
    [1.2, 1.1], [1.4, 0.8], [2.0, 2.2], [2.2, 1.9],
    [2.4, 2.5], [2.8, 2.1], [3.0, 2.7], [3.2, 2.4]
])
y = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.33, random_state=7, stratify=y
)

for C in [0.01, 1.0, 100.0]:
    clf = make_pipeline(StandardScaler(), LogisticRegression(C=C, solver="liblinear"))
    clf.fit(X_train, y_train)
    train_acc = accuracy_score(y_train, clf.predict(X_train))
    val_acc = accuracy_score(y_val, clf.predict(X_val))
    print(f"C={C:>6} train_accuracy={train_acc:.2f} validation_accuracy={val_acc:.2f}")

The deterministic output format reports training and validation accuracy for three C values. Small C applies strong regularization and may underfit. Large C weakens the penalty and can overfit if there are many features, separable classes, or noisy labels. The best value is the one that survives validation, not the one that makes coefficients largest.

Example 4: Reading a Learning Curve

A learning curve trains the same modeling pipeline on increasing amounts of data. It is one of the clearest diagnostics for deciding whether regularization or more data is likely to help.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import learning_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_regression(n_samples=80, n_features=12, noise=18.0, random_state=4)
model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
train_sizes, train_scores, validation_scores = learning_curve(
    model, X, y, cv=5, scoring="neg_root_mean_squared_error",
    train_sizes=[0.25, 0.5, 0.75, 1.0]
)

for size, train_score, validation_score in zip(train_sizes, train_scores, validation_scores):
    train_rmse = -train_score.mean()
    validation_rmse = -validation_score.mean()
    print(f"rows={size:2d} train_rmse={train_rmse:6.1f} validation_rmse={validation_rmse:6.1f}")

If training RMSE is low and validation RMSE is much higher, the model has high variance. More relevant data, stronger regularization, or simpler features may help. If both errors are high and close together, the model is underfitting; stronger regularization would probably make it worse. If the validation curve keeps improving as rows increase, collecting more data is a plausible next investment.

Design Choices and Trade-offs

Choosing a regularizer is a modeling decision, not a checkbox. L2 is a strong default for dense numeric features because it is stable and keeps correlated signals. L1 is useful when you expect only a small subset of features to matter or when a sparse model is easier to inspect. Elastic net is helpful when you want sparsity but have groups of correlated features, such as similar text terms or related sensor measurements.

Simplifying the representation can be better than increasing a penalty. Dropping a leaky timestamp, reducing polynomial degree, limiting text vocabulary, or grouping rare categories can reduce variance before optimization begins. More data is most useful when it is relevant to the deployment distribution. More rows from the same biased source may reinforce the wrong pattern. Data augmentation can regularize image, audio, and text models when the transformations preserve labels, but harmful augmentation teaches invariances that are not true.

There is also an interpretability trade-off. A heavily regularized linear model may be easier to explain than a deep tree ensemble, but coefficients after scaling, correlated inputs, or target encoding still require care. Regularization improves generalization; it does not automatically make a model causal, fair, or robust outside the data distribution.

Failure Modes and Troubleshooting

Symptom: training accuracy is near perfect while validation accuracy is poor. Cause: the model capacity is too high for the amount and quality of data, or the features include noise that happens to align with the training labels. Diagnostic steps: compare training and validation metrics, run cross-validation, inspect feature counts, and test a simpler baseline. Correction: increase regularization, reduce feature complexity, collect more representative data, or use early stopping.

Symptom: validation score looked excellent during development but fails on the test set. Cause: repeated tuning has overfit the validation set, or preprocessing was fit before splitting. Diagnostic steps: review the pipeline, confirm that scalers and encoders are inside cross-validation, and count how many times the validation set influenced decisions. Correction: restore a fresh holdout set, use nested cross-validation for model selection, and freeze the final evaluation protocol.

Symptom: stronger regularization makes every metric worse. Cause: the model may already be underfitting, the features may be too weak, or the penalty scale may be inappropriate because inputs were not standardized. Diagnostic steps: inspect learning curves, compare against a low-regularization model, and check feature scales. Correction: standardize numeric features, lower the penalty, add informative features, or choose a model class with enough capacity.

Security, Performance, and Reliability

Regularization has reliability implications because it reduces sensitivity to individual records. That can make predictions less brittle when new data contains small measurement shifts. It is not a privacy guarantee, but overfit models are more likely to memorize rare examples, so regularization, deduplication, and careful evaluation can reduce memorization risk. For sensitive datasets, combine these practices with access controls, minimization, and privacy-specific methods when needed.

Performance trade-offs are mixed. Simpler models and sparse lasso solutions can be faster at inference. Cross-validation and hyperparameter search cost more during training because the model is fit many times. Limit the search space deliberately: test logarithmic grids such as 0.001, 0.01, 0.1, 1, 10, and stop expanding once validation behavior is clear.

Hands-on Lab: Tune Regularization

Prerequisites: Python with NumPy and scikit-learn installed, and a terminal or notebook where you can run short scripts. The lab uses synthetic data so there is no external dataset to download.

  1. Create a regression dataset with make_regression, using more features than are truly needed and a nonzero noise value.
  2. Build a pipeline containing StandardScaler and Ridge. Scaling is required because the L2 penalty acts on coefficient size, and coefficient size depends on feature scale.
  3. Evaluate alpha values such as 0.001, 0.01, 0.1, 1, 10, and 100 with five-fold cross-validation using negative root mean squared error.
  4. Record the mean and standard deviation of validation RMSE for each alpha.
  5. Select the alpha with the lowest mean validation RMSE, then fit one final pipeline on the training split only.
  6. Verify on a holdout test split that was not used to choose alpha.

Verification: your selected alpha should not be justified by training error alone. You should be able to show a small table where validation RMSE initially improves as alpha reduces variance, then worsens once the model becomes too constrained. Cleanup: remove temporary notebooks or scripts and reset any random seeds only if your project standard requires nondeterministic training elsewhere.

Assessment Exercises

  • A model has 0.99 training accuracy and 0.71 validation accuracy. Name two plausible causes and describe one experiment that distinguishes them.
  • You add L2 regularization to a linear model and performance collapses. Why should feature scaling be one of the first checks?
  • In a text classifier with 200,000 n-gram features, when might L1 or elastic net be preferable to pure L2?
  • Explain why choosing hyperparameters after looking repeatedly at the test set is another form of overfitting.
  • Given a learning curve where training and validation errors are both high and close, should you increase regularization? Defend your answer.

Summary

Overfitting is the gap between memorizing the training sample and learning a pattern that holds on future data. Regularization changes the training objective or model design so simpler explanations are preferred unless complexity earns its keep on validation data. Use learning curves, cross-validation, pipelines, and final holdout tests to separate true improvement from training-set or validation-set luck.

Progress is saved in this browser.