Decision Trees and Random Forests

Decision trees and random forests are supervised learning models for turning feature values into class labels or numeric predictions. A tree learns a sequence of questions such as petal_width <= 0.8; a forest learns many different trees and averages their votes. By the end of this lesson you should be able to read a tree path, explain why a split was chosen, tune depth and leaf size, and diagnose common failures such as overfitting, leakage, unstable feature importance, and poor performance on rare cases.

What a Tree Learns

A decision tree partitions the feature space into rectangles. At each internal node, the training algorithm searches for one feature and one threshold that make the child nodes purer than the parent. In classification, purity often means low Gini impurity or low entropy. In regression, it usually means lower squared error or absolute error. A leaf stores the prediction used for future rows that land in that region: class proportions for classification or an average target value for regression.

Consider a binary classifier. Gini impurity is 1 - sum(p_k^2), where p_k is the fraction of class k in a node. A node containing five positive and five negative examples has Gini 0.5. A node containing nine positives and one negative has Gini 0.18. The algorithm computes the weighted impurity after many candidate splits and chooses the split with the largest impurity reduction. This greedy choice is local: it does not prove the globally best tree, but it is efficient and works well with regularization.

Tree Anatomy and API

The important vocabulary is concrete. The root is the first split. A node contains the training rows reaching that point. A threshold compares one numeric feature against a value. A leaf has no further split and emits a prediction. max_depth limits how many decisions a row can pass through. min_samples_leaf prevents tiny leaves. criterion selects the impurity measure. class_weight changes the cost of mistakes when classes are imbalanced.

Random forests add two more mechanisms. First, each tree is fitted on a bootstrap sample: a same-size sample drawn from the training rows with replacement. Some rows are repeated; some are left out. Second, at each split the tree considers only a random subset of features. Bagging reduces variance because the trees make different errors, and feature subsampling keeps strong predictors from dominating every tree. Prediction then aggregates: classification uses majority vote or averaged class probabilities, while regression uses the mean of tree predictions.

Example 1: A Transparent Classification Tree

This example trains a shallow tree on the iris data. The model is intentionally small so the learned rules can be inspected. Expected behavior is deterministic because the tree has a fixed random seed and a fixed depth.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
X = iris.data
y = iris.target

model = DecisionTreeClassifier(max_depth=2, random_state=7)
model.fit(X, y)

print(export_text(model, feature_names=list(iris.feature_names)))
print(model.predict([[5.1, 3.5, 1.4, 0.2]]).tolist())

The printed tree shows nested tests and the final prediction is [0], the setosa class. The first split is usually on petal width because that feature separates setosa from the other classes cleanly. The important habit is to read a prediction as a path, not as a mysterious score: the sample travels from root to leaf by satisfying one threshold at a time.

Example 2: Depth, Leaves, and Overfitting

A fully grown tree can memorize noisy training data. The following code compares a deep tree with a restricted tree on a synthetic classification problem. The exact scores can vary with library details, but the pattern is stable: the unconstrained tree is excellent on training rows and less reliable on held-out rows.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(
    n_samples=800,
    n_features=8,
    n_informative=4,
    n_redundant=0,
    flip_y=0.08,
    random_state=11,
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=11, stratify=y
)

memorizer = DecisionTreeClassifier(random_state=11)
regularized = DecisionTreeClassifier(max_depth=4, min_samples_leaf=12, random_state=11)

for name, model in [("memorizer", memorizer), ("regularized", regularized)]:
    model.fit(X_train, y_train)
    print(name, round(model.score(X_train, y_train), 3), round(model.score(X_test, y_test), 3))

If the first number is much higher than the second, the model has high variance. Correction usually starts with max_depth, min_samples_leaf, min_samples_split, or pruning through ccp_alpha. These parameters reduce how many narrow regions the tree can create, forcing leaves to represent patterns shared by more observations.

Example 3: A Random Forest for Robustness

A random forest often improves held-out performance without making one tree deeper. Each tree sees a different bootstrap sample and a different subset of candidate features, so the ensemble smooths out brittle splits.

from sklearn.datasets import make_regression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X, y = make_regression(
    n_samples=600,
    n_features=6,
    n_informative=4,
    noise=18.0,
    random_state=4,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=4)

forest = RandomForestRegressor(
    n_estimators=200,
    max_features="sqrt",
    min_samples_leaf=4,
    random_state=4,
    n_jobs=-1,
)
forest.fit(X_train, y_train)
predictions = forest.predict(X_test)
print(round(mean_absolute_error(y_test, predictions), 2))
print([round(v, 3) for v in forest.feature_importances_])

The first line is the mean absolute error on held-out data. The second line lists impurity-based feature importances, which are useful for exploration but not a final causal explanation. Correlated features can split credit, and high-cardinality features can look too important. When interpretation matters, compare with permutation importance on a validation set.

Design Choices and Trade-offs

Decision trees are easy to explain and require little preprocessing. They handle nonlinear interactions, mixed scales, and monotonic threshold rules naturally. They do not require standardization, and they can expose simple decision paths to analysts. Their weakness is instability: a small data change can move an early split and alter many downstream leaves. Deep trees also extrapolate poorly because each leaf predicts from observed training targets.

Random forests trade interpretability for accuracy and stability. A forest is less likely to overfit a single odd split, but hundreds of trees are harder to inspect. Larger n_estimators usually reduces variance until returns flatten, while increasing training and prediction cost. Smaller max_features makes trees more diverse but can ignore useful predictors at some splits. Larger min_samples_leaf smooths predictions but may miss rare but real segments. Out-of-bag scoring can estimate generalization from bootstrap leftovers, but a separate test set is still cleaner for final reporting.

Failure Modes and Troubleshooting

Symptom: training accuracy is near perfect while validation performance is poor. Cause: the tree has memorized noise or duplicate patterns. Diagnose: compare train and validation metrics, inspect depth and leaf sample counts, and check whether leaves contain one or two samples. Correct: increase min_samples_leaf, set max_depth, use cross-validation, or move to a forest.

Symptom: validation scores look excellent, but live predictions fail. Cause: target leakage, often through features computed after the outcome. Diagnose: audit each feature for availability at prediction time and retrain without suspicious columns. Correct: build feature pipelines around prediction-time data only and split data by time when future information could leak backward.

Symptom: minority class recall is poor even with high overall accuracy. Cause: impurity reduction favors the majority class when rare cases are not weighted. Diagnose: use a confusion matrix, precision-recall metrics, and stratified splits. Correct: set class weights, tune probability thresholds, collect more rare examples, or evaluate with recall and precision rather than accuracy alone.

Symptom: feature importance changes between runs. Cause: correlated inputs, small data, or too few trees. Diagnose: increase tree count, run repeated fits, and compare impurity importance with permutation importance. Correct: group correlated features, report uncertainty, and avoid treating importance as proof of causality.

Reliability and Performance Implications

Tree models are usually safe from numeric scaling problems, but they can become large. A deep forest can consume memory, slow batch scoring, and make model artifacts expensive to move. Constrain tree size when latency matters, measure prediction time at realistic batch sizes, and pin the feature order through a pipeline so training and inference columns cannot drift. For security and privacy, avoid logging raw sensitive features during diagnostics, and treat serialized model files as executable-adjacent artifacts: load them only from trusted storage because many Python serialization formats can execute code during loading.

Hands-on Lab

Prerequisites: Python with scikit-learn installed, a terminal, and permission to run local scripts. The lab uses built-in datasets only.

  1. Create a script named tree_forest_lab.py.
  2. Load a classification dataset and split it with stratification.
  3. Train one shallow decision tree and one random forest.
  4. Print accuracy, macro F1, a confusion matrix, and the depth of the single tree.
  5. Change min_samples_leaf from 1 to 10 and rerun.
  6. Verify that the larger leaf size usually lowers training fit and can improve validation behavior when the original tree overfit.
  7. Cleanup by deleting the script if it was only for practice; no external resources are created.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=21, stratify=y
)

models = {
    "tree": DecisionTreeClassifier(max_depth=5, min_samples_leaf=10, random_state=21),
    "forest": RandomForestClassifier(
        n_estimators=150, max_features="sqrt", min_samples_leaf=4, random_state=21
    ),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    predicted = model.predict(X_test)
    print(name)
    print("accuracy", round(accuracy_score(y_test, predicted), 3))
    print("macro_f1", round(f1_score(y_test, predicted, average="macro"), 3))
    print(confusion_matrix(y_test, predicted).tolist())

print("tree_depth", models["tree"].get_depth())

Verification is not a single magic score. You should see both models print metrics and a 2-by-2 confusion matrix. The forest should usually be competitive with or better than the constrained tree, but the lesson is the workflow: compare on held-out data, inspect errors, and change one control at a time.

Assessment Exercises

  • A tree has 99% training accuracy and 78% validation accuracy. Which hyperparameters would you adjust first, and what evidence would confirm the change helped?
  • Explain why a random forest with max_features="sqrt" can outperform a forest where every split sees every feature.
  • You discover a feature called days_until_cancellation in a churn model. Why is this suspicious, and how would you test for leakage?
  • Given two equally accurate models, one shallow tree and one large forest, choose one for a regulated manual review workflow and justify the trade-off.
  • Design a validation split for transaction fraud data where behavior changes over time. Why might a random split be misleading?

Summary

A decision tree learns greedy threshold splits that reduce impurity and stores predictions in leaves. Its strengths are readability and flexible nonlinear rules; its risks are instability and overfitting. A random forest builds many randomized trees on bootstrap samples and aggregates them to reduce variance. In supervised learning work, use trees to understand rule structure, use forests for stronger generalization, and validate both with leakage checks, class-sensitive metrics, and realistic held-out data.