Module 5 · Improving Models

Hyperparameter Tuning

Lesson 15 of 24 10 min read Updated

Hyperparameter tuning is the process of choosing model settings that are not learned directly from training examples. A linear model learns coefficients, a tree learns split rules, and a neural network learns weights; those are parameters. Hyperparameters are choices such as regularization strength, tree depth, learning rate, number of neighbors, batch size, and feature preprocessing options. They shape how learning happens.

The practical outcome is not to find the setting that looks best once. It is to choose a setting that is likely to generalize to new data. That is why tuning belongs in the Improving Models section: after you have a working baseline, you use validation data to control underfitting, overfitting, runtime, and model complexity without accidentally training on the final test set.

How Tuning Works Internally

A tuning run has four moving parts: a search space, a splitting strategy, a scoring function, and a selection rule. The search space defines candidate values, such as C values for logistic regression or max_depth values for a decision tree. The splitting strategy decides which rows are used for fitting and which rows are used for validation. The scoring function turns each fitted candidate into a number. The selection rule chooses the candidate with the best average validation score, often breaking ties with a simpler or faster model.

The important detail is that each candidate must be evaluated on data it did not fit. In k-fold cross-validation, the training data is divided into k parts. For each candidate, the model is fitted k times. Each time, one fold is held out for validation and the remaining folds are used for fitting. The candidate receives k validation scores, and the tuner compares candidates by their mean score, sometimes also inspecting score variance.

The final test set has a different job. It should not participate in choosing hyperparameters. If you evaluate many settings on the test set and pick the best one, the test set has become a validation set. The reported score will usually be too optimistic because the choice adapted to random quirks in that data.

API Anatomy

Most Python tuning APIs contain the same concepts even when names differ. In scikit-learn, an estimator supplies fit and predict methods. A parameter grid or distribution names hyperparameters using estimator field names, or pipeline step prefixes such as model__C. A cross-validation object controls folds. A scorer defines whether larger is better. The tuning object stores best_params_, best_score_, and a table of per-candidate results.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=300, n_features=8, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=7
)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000, solver="liblinear")),
])

search = GridSearchCV(
    estimator=pipe,
    param_grid={"model__C": [0.01, 0.1, 1.0, 10.0]},
    scoring="accuracy",
    cv=5,
)
search.fit(X_train, y_train)

print(search.best_params_)
print(round(search.best_score_, 3))
print(round(search.score(X_test, y_test), 3))

This example tunes regularization strength. Smaller C means stronger regularization, which can reduce variance but increase bias. The printed scores are deterministic for the fixed random seed, though exact values can vary slightly across library implementations. The important behavior is that the best setting is selected from cross-validation on X_train, then scored once on X_test.

Example 1: Manual Validation Split

A single validation split is the simplest tuning pattern. It is fast and easy to reason about, but the result can depend heavily on which rows land in validation.

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X, y = make_regression(n_samples=200, n_features=6, noise=15.0, random_state=3)
X_train, X_valid, y_train, y_valid = train_test_split(
    X, y, test_size=0.25, random_state=3
)

results = []
for alpha in [0.01, 0.1, 1.0, 10.0, 100.0]:
    model = Ridge(alpha=alpha)
    model.fit(X_train, y_train)
    predictions = model.predict(X_valid)
    mae = mean_absolute_error(y_valid, predictions)
    results.append((mae, alpha))

best_mae, best_alpha = min(results)
print(f"best alpha={best_alpha}, validation MAE={best_mae:.2f}")

The candidate with the lowest mean absolute error wins. This pattern is useful for a quick baseline or a large dataset where cross-validation is too expensive. Its weakness is uncertainty: one unlucky split can favor a setting that would not be best on other representative samples.

Example 2: Cross-Validated Grid Search

Grid search evaluates every combination in a discrete grid. It is predictable and easy to audit, which makes it a good default when there are only a few hyperparameters.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=11
)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", SVC()),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=11)
search = GridSearchCV(
    pipe,
    {
        "model__kernel": ["linear", "rbf"],
        "model__C": [0.1, 1.0, 10.0],
        "model__gamma": ["scale", 0.01, 0.1],
    },
    scoring="f1",
    cv=cv,
)
search.fit(X_train, y_train)

print(search.best_params_)
print(f"cv f1={search.best_score_:.3f}")
print(f"test f1={search.score(X_test, y_test):.3f}")

The pipeline matters. Scaling is fitted inside each training fold, then applied to that fold’s validation rows. If you scaled the whole dataset before cross-validation, the validation folds would leak information from their own distributions into preprocessing.

Example 3: Random Search for Wider Spaces

Random search samples from distributions instead of trying every combination. It often finds strong settings faster when only some hyperparameters matter or when useful values span orders of magnitude.

from scipy.stats import loguniform
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV, train_test_split

X, y = make_classification(
    n_samples=800, n_features=20, n_informative=8, random_state=21
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=21
)

search = RandomizedSearchCV(
    RandomForestClassifier(random_state=21),
    {
        "n_estimators": [100, 200, 400],
        "max_depth": [None, 4, 8, 16],
        "min_samples_leaf": [1, 2, 5, 10],
        "max_features": loguniform(0.1, 1.0),
    },
    n_iter=12,
    scoring="roc_auc",
    cv=4,
    random_state=21,
)
search.fit(X_train, y_train)

print(search.best_params_)
print(f"cv auc={search.best_score_:.3f}")
print(f"test auc={search.score(X_test, y_test):.3f}")

Here the search budget is n_iter=12. Increasing it usually improves the chance of finding a better setting but increases runtime. A log-uniform distribution is appropriate for values where multiplying by ten is more meaningful than adding a fixed amount.

Design Choices and Trade-Offs

Grid search is exhaustive over the values you list, but it becomes expensive as dimensions multiply. Random search covers broader spaces for a fixed budget, but it may miss a narrow optimum. Bayesian optimization and successive halving methods can be more sample-efficient, but they add complexity and can be harder to reproduce or explain.

The metric is a modeling decision, not a decoration. Accuracy can hide poor minority-class performance. F1 balances precision and recall but ignores probability calibration. ROC AUC measures ranking quality across thresholds. Mean absolute error is easier to explain than root mean squared error, while root mean squared error punishes large misses more strongly.

Cross-validation also has variants. Use stratified folds for classification with uneven classes. Use grouped folds when rows from the same patient, customer, device, or document must not appear in both training and validation. Use time-series splits when future rows must not help predict past rows.

Failure Modes and Troubleshooting

Symptom: validation score is excellent, but the test score collapses. Cause: leakage, repeated test-set peeking, or a validation split that does not match deployment data. Diagnose: inspect preprocessing order, check whether duplicate or related records cross folds, and compare train, validation, and test distributions. Correct: move preprocessing into a pipeline, use grouped or time-aware splitting, and reserve a fresh test set if the old one drove many decisions.

Symptom: every candidate performs nearly the same. Cause: the search space may be irrelevant, too narrow, or dominated by a preprocessing bottleneck. Diagnose: review cv_results_, compare against a simple baseline, and plot scores by hyperparameter. Correct: widen meaningful ranges, tune preprocessing choices, or change the model family.

Symptom: tuning takes too long or exhausts memory. Cause: too many combinations, large folds, expensive estimators, or excessive parallel jobs. Diagnose: compute candidates times folds, profile one fit, and monitor memory per worker. Correct: reduce the grid, use random search, lower cv, cache pipeline transforms, or tune on a representative sample before confirming on the full training set.

Reliability and Performance Implications

Tuning can silently create unreliable models if experiments are not reproducible. Set random seeds for splits and stochastic estimators when comparing candidates. Store the dataset version, feature code version, search space, metric, fold strategy, and selected parameters. Without those records, you cannot explain why a model changed.

Performance costs are multiplicative. A grid of 40 candidates with 5-fold cross-validation fits 200 models before refitting the final one. Parallelism can reduce wall-clock time, but each worker may copy data and model state. Treat tuning as a resource-budgeted experiment, not an unbounded loop.

Hands-On Lab

Prerequisites: Python with scikit-learn and scipy installed, and a terminal or notebook where you can run a single script. The lab goal is to tune a classifier without leaking validation information.

  1. Create a train/test split with stratification and a fixed random seed.
  2. Build a pipeline containing scaling and a support-vector classifier.
  3. Run GridSearchCV over C, kernel, and gamma.
  4. Print the best parameters, best cross-validation score, and final test score.
  5. Verify that scaling is inside the pipeline and that X_test is used only in the final score call.
  6. Cleanup is simple: remove any temporary script or notebook outputs. If you logged experiments, mark this run as a learning run rather than a production candidate.

Verification should focus on behavior: the search object should report a best parameter dictionary, the cross-validation score should be based only on training folds, and the test score should be computed once after selection.

Assessment Exercises

  • You tune 100 configurations and choose the one with the best test-set score. Explain why the reported test score is no longer an honest estimate.
  • A dataset contains multiple rows per customer. Which cross-validation strategy should you consider, and what failure does it prevent?
  • For a fraud classifier with rare positives, compare accuracy, F1, and ROC AUC as tuning metrics. Which would you inspect first and why?
  • A grid search with 6 values for one parameter, 5 for another, 4 for another, and 5 folds is planned. How many model fits are required before the final refit?
  • Describe one reason random search may beat grid search under a fixed runtime budget.

Summary

Hyperparameter tuning chooses learning settings by measuring candidate models on held-out validation data. The core discipline is separation: training folds fit models, validation folds select settings, and the test set estimates final generalization. Strong tuning practice uses pipelines to avoid leakage, search spaces that reflect model behavior, metrics that match the problem, and records detailed enough to reproduce the decision.

Progress is saved in this browser.