Linear Regression

Linear regression predicts a numeric outcome by fitting a straight-line relationship between features and a target. In this supervised learning section, it is the baseline model you should understand before using more flexible estimators: it exposes how loss functions, fitted parameters, residuals, train/test evaluation, and feature design affect a model’s behavior.

Purpose and Outcome

The purpose is not merely to draw a line through points. A fitted linear regression model gives you an equation: prediction equals an intercept plus one coefficient for each feature. That equation can estimate a future value, quantify the direction and size of each feature’s association, and reveal whether the chosen features describe the target well enough for the task. By the end of this lesson, you should be able to fit a small regression by hand or in Python, interpret coefficients carefully, inspect residuals, and recognize when a linear model is too simple or numerically fragile.

Mechanism: Least Squares Internals

For one feature, the model is y_hat = b0 + b1*x. The intercept b0 is the prediction when the feature is zero; the slope b1 is the predicted change in the target for a one-unit increase in the feature. For multiple features, the same idea becomes y_hat = b0 + b1*x1 + b2*x2 + .... Training chooses the coefficients that minimize the sum of squared residuals, where a residual is actual - predicted.

Squaring residuals matters. It makes positive and negative errors add instead of canceling, penalizes large misses more heavily than small misses, and gives a smooth objective with a closed-form solution when the design matrix is well behaved. In matrix notation, predictions are X beta, residuals are y - X beta, and ordinary least squares solves the normal equation X'X beta = X'y. Many libraries avoid directly inverting X'X and use more stable decompositions, but the normal equation is still useful for understanding what is being estimated.

The fitted line always balances residuals according to the model space you gave it. If you omit an important feature, include duplicate features, mix incompatible units, or force a straight line onto curved data, the optimizer still returns the best straight-line compromise. The important question is whether that compromise is valid for the prediction or explanation you need.

API Anatomy

A regression workflow has four parts. First, define the target as a continuous numeric value measured after the features are known. Second, build a feature matrix: one row per observation and one column per feature. Third, fit the estimator on training data only. Fourth, evaluate predictions on data not used for fitting with metrics such as mean absolute error, root mean squared error, and residual plots.

In Python libraries, the common API is fit(X_train, y_train), then predict(X_test). The shape of X is usually two-dimensional even for one feature. Preprocessing, such as imputation, scaling, one-hot encoding, and polynomial expansion, should be fitted on the training split and then applied to validation or test splits. That separation prevents information from the evaluation data from leaking into the fitted coefficients.

Example 1: One Feature

Suppose you predict exam score from study hours. The following code computes the least-squares intercept and slope directly, so the arithmetic is visible.

from statistics import mean

def fit_simple_linear_regression(x, y):
    if len(x) != len(y) or len(x) < 2:
        raise ValueError("x and y must have the same length of at least 2")
    x_bar = mean(x)
    y_bar = mean(y)
    numerator = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y))
    denominator = sum((xi - x_bar) ** 2 for xi in x)
    if denominator == 0:
        raise ValueError("x must vary")
    slope = numerator / denominator
    intercept = y_bar - slope * x_bar
    return intercept, slope

hours = [1, 2, 3, 4, 5]
scores = [52, 56, 61, 65, 70]
intercept, slope = fit_simple_linear_regression(hours, scores)
prediction = intercept + slope * 6
print(f"intercept={intercept:.2f}, slope={slope:.2f}")
print(f"predicted score for 6 hours={prediction:.1f}")

The output is deterministic: intercept=47.30, slope=4.50 and predicted score for 6 hours=74.3. The slope says each additional study hour is associated with about 4.5 more score points in this tiny dataset. It does not prove that forcing a student to study one extra hour causes exactly 4.5 points of improvement.

Example 2: Multiple Features

Real problems often need more than one signal. This example estimates a house price from size and bedroom count by solving the normal equation with Gaussian elimination.

def transpose(matrix):
    return [list(column) for column in zip(*matrix)]

def matmul(a, b):
    return [[sum(x * y for x, y in zip(row, col)) for col in zip(*b)] for row in a]

def solve_linear_system(a, b):
    n = len(a)
    aug = [row[:] + [value] for row, value in zip(a, b)]
    for col in range(n):
        pivot = max(range(col, n), key=lambda r: abs(aug[r][col]))
        if abs(aug[pivot][col]) < 1e-12:
            raise ValueError("singular design matrix")
        aug[col], aug[pivot] = aug[pivot], aug[col]
        scale = aug[col][col]
        aug[col] = [value / scale for value in aug[col]]
        for row in range(n):
            if row == col:
                continue
            factor = aug[row][col]
            aug[row] = [current - factor * base for current, base in zip(aug[row], aug[col])]
    return [row[-1] for row in aug]

def fit_linear_regression(features, target):
    design = [[1.0] + row for row in features]
    xt = transpose(design)
    xtx = matmul(xt, design)
    xty = [sum(row[i] * target[i] for i in range(len(target))) for row in xt]
    return solve_linear_system(xtx, xty)

features = [[900, 1], [1100, 1], [1300, 2], [1500, 2], [1700, 3]]
prices = [220000, 245000, 305000, 330000, 390000]
coef = fit_linear_regression(features, prices)
size, bedrooms = 1400, 2
estimate = coef[0] + coef[1] * size + coef[2] * bedrooms
print([round(value, 2) for value in coef])
print(f"estimated price=${estimate:,.0f}")

The coefficients are [72500.0, 125.0, 35000.0], and the estimate is $317,500. Holding bedroom count fixed, the model assigns about 125 dollars per square foot. Holding size fixed, it assigns about 35,000 dollars per bedroom. With so few rows, this is a teaching example rather than a credible housing model, but it shows how each column receives its own coefficient.

Example 3: Residual Diagnostics

A low average error can hide structure. Here the target grows faster at high advertising spend, so a straight line leaves a pattern in the residuals.

from statistics import mean

def fit_simple_linear_regression(x, y):
    x_bar = mean(x)
    y_bar = mean(y)
    slope = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y)) / sum((xi - x_bar) ** 2 for xi in x)
    intercept = y_bar - slope * x_bar
    return intercept, slope

def predict(intercept, slope, x):
    return [intercept + slope * value for value in x]

def residual_report(actual, predicted):
    residuals = [a - p for a, p in zip(actual, predicted)]
    mae = sum(abs(r) for r in residuals) / len(residuals)
    bias = sum(residuals) / len(residuals)
    return residuals, mae, bias

ad_spend = [1, 2, 3, 4, 5, 6]
sales = [9, 12, 15, 19, 26, 38]
intercept, slope = fit_simple_linear_regression(ad_spend, sales)
fitted = predict(intercept, slope, ad_spend)
residuals, mae, bias = residual_report(sales, fitted)
print(f"slope={slope:.2f}, MAE={mae:.2f}, bias={bias:.2f}")
print([round(value, 2) for value in residuals])

The output is slope=5.46, MAE=2.56, bias=0.00 and residuals [2.81, 0.35, -2.1, -3.56, -2.02, 4.52]. The near-zero bias is expected because ordinary least squares with an intercept balances residuals around zero. The curved residual pattern is the warning: early and late points are underpredicted while middle points are overpredicted. A polynomial feature, log transform, or different model may fit the mechanism better.

Design Choices and Trade-offs

Linear regression is fast, transparent, and data efficient. It is a strong default when you need an interpretable baseline or when each feature plausibly contributes an additive effect. Its weaknesses follow from the same simplicity. It extrapolates linearly outside the observed range, is sensitive to outliers because errors are squared, and cannot represent interactions or curvature unless you add suitable features.

Feature scaling does not change ordinary least-squares predictions, but it changes coefficient magnitudes and improves numerical behavior for regularized variants. Collinearity, where columns carry nearly duplicate information, can make coefficients unstable even when predictions look reasonable. Regularization addresses that by adding a penalty: ridge regression shrinks coefficients smoothly, while lasso can drive some coefficients to zero. Those are still linear models, but the training objective changes from pure residual minimization to residual minimization plus a complexity penalty.

Failure Modes and Troubleshooting

If training fails with a singular matrix error, the symptoms are coefficients that cannot be computed or library warnings about rank deficiency. The usual cause is duplicate columns, a constant feature, or more features than independent observations. Diagnose by checking column variance, correlations, and the rank of the design matrix. Correct it by removing redundant features, collecting more independent data, or using ridge regression.

If test error is much worse than training error, suspect overfitting, leakage in validation design, or a train/test split that does not match deployment. Reproduce the split, compare feature distributions, and verify that preprocessing is fitted only on training data. If residuals fan out as predictions grow, the constant-variance assumption is weak; inspect residuals by predicted value, try transforming the target, or use a metric aligned to relative error.

If a coefficient has the wrong sign, do not immediately assume the model is broken. Correlated features can change coefficient interpretation because each coefficient is conditional on the others. Check pairwise correlations, refit smaller models, inspect units, and confirm that target timing does not allow future information to leak backward.

Reliability and Performance Implications

Linear regression is computationally cheap at prediction time: one multiplication per feature plus additions. Training cost depends on the solver and feature count; very wide matrices can still be expensive or unstable. Reliability depends more on data discipline than on infrastructure. Store the feature schema with the model, reject missing or reordered columns, and monitor residual distributions after deployment. For sensitive domains, avoid treating coefficients as causal explanations unless the data collection design supports causal inference.

Hands-on Lab

Prerequisites: Python 3 and a terminal. Create a temporary directory and place the three code examples into separate files named simple.py, multiple.py, and residuals.py. Step 1: run python simple.py and confirm the predicted score for six hours is 74.3. Step 2: run python multiple.py and confirm the price estimate is $317,500. Step 3: run python residuals.py and inspect the residual list. Step 4: change the final sales value from 38 to 31 and rerun the residual example; the late positive residual should shrink, showing how outliers influence the fitted slope and error.

Verification is successful when each script prints the expected deterministic values before your change, and when the changed residual example produces a different slope. Cleanup is simply deleting the temporary directory. No persistent service, credentials, or external dataset is required.

Assessment Exercises

• A dataset has two identical feature columns. Explain why ordinary least squares cannot uniquely assign separate coefficients to them, and name one correction.
• You add a feature that was recorded after the target event. Why can validation error improve while the deployed model becomes unusable?
• In the advertising example, what residual pattern suggests that a straight line is missing curvature?
• When would a less accurate linear model be preferable to a more accurate black-box model?
• Design a train/test split for home prices when the model will be used next quarter in a city whose prices are rising.

Summary

Linear regression fits an additive equation by minimizing squared residuals. Its value in machine learning is both practical and diagnostic: it gives a quick baseline, exposes feature and target choices, and makes residual problems visible. Use it when a transparent linear approximation is acceptable, inspect residuals before trusting it, and move to transformed features, regularization, or nonlinear models when the data shows that the straight-line assumptions are not holding.