Correlation, Confounding, and Interpretable Regression

Module 6: Statistics and Sound Decisions · Lecture 24 of 32

A strong association can arise because two variables share another cause. Use a transparent synthetic construction to compare a raw relationship with a conditional regression and understand why adjustment still does not automatically prove causation.

What you will learn

Calculate a correlation and fit an ordinary least-squares model.
Recognize a shared-factor explanation for association.
Check design-matrix rank and avoid causal overclaims.

Understand the problem

Association is a description

Correlation summarizes a linear relationship in observed values. It can miss nonlinear patterns, react strongly to unusual points, and combine distinct subpopulations into a misleading overall summary. Always inspect the variables' meanings, units, and collection process before interpreting a coefficient.

Construct a confounder

In this simulation, an underlying factor z influences both x and y. The data-generating equation does not include a direct x effect on y. Nevertheless, x and y can have a strong raw correlation because both vary with z. This is an educational construction with known rules; real causal structures are not usually known so conveniently.

Fit a conditional linear model

A design matrix contains an intercept and selected explanatory columns. NumPy least squares finds coefficients that minimize squared residuals. Adding z can reduce the x coefficient in this construction because the model now distinguishes their information. Check rank: perfectly redundant columns make individual coefficients non-identifiable. Coefficient units matter; a one-unit increase in x has no meaning until x's scale is declared.

Adjustment has assumptions

Controlling for a variable does not automatically remove bias. Unmeasured confounders, measurement error, wrong functional form, selection, or conditioning on an inappropriate variable can still mislead. A causal claim needs a defensible design and assumptions, not merely a regression package. For prediction, evaluation on genuinely held-out data is also required; that belongs to the site's machine-learning course.

Run the worked example

Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 24 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.

from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT

rng = np.random.default_rng(24)
z = rng.normal(size=1000)
x = z + rng.normal(scale=0.3, size=1000)
y = 10 * z + rng.normal(scale=0.8, size=1000)
simple_design = np.column_stack([np.ones(len(x)), x])
adjusted_design = np.column_stack([np.ones(len(x)), x, z])
simple_coef = np.linalg.lstsq(simple_design, y, rcond=None)[0]
adjusted_coef = np.linalg.lstsq(adjusted_design, y, rcond=None)[0]
correlation = np.corrcoef(x, y)[0, 1]
print('Raw correlation:', round(float(correlation), 4))
print('Simple x slope:', round(float(simple_coef[1]), 4))
print('Adjusted x slope:', round(float(adjusted_coef[1]), 4))
assert np.linalg.matrix_rank(adjusted_design) == 3
assert abs(adjusted_coef[1]) < abs(simple_coef[1])

Verified output

Raw correlation: 0.9598
Simple x slope: 9.3366
Adjusted x slope: 0.0531

Walk through the result

The raw correlation is strong because x and y share z. The simple regression assigns much of that shared variation to x. Once z enters the model, the x coefficient becomes much smaller in this particular realization. That result agrees with the known simulation equation, but the logic must not be reversed into “a small adjusted coefficient proves no causal effect” for arbitrary observational data.

Common mistakes and diagnosis

Do not mix correlations across incompatible populations without checking subgroup structure. Do not interpret a coefficient while ignoring its units or the other included variables. Perfect collinearity cannot be fixed by collecting more identical copies of the same information. A high in-sample fit can result from leakage or overfitting. Do not describe association as impact, influence, or incremental return without a design that supports those words.

Practice assignment

Add a duplicate of x as another design column and show that matrix rank is lower than the number of columns. Then compare mean squared residuals for the simple and adjusted fits. Explain why lower training error alone is not evidence of a better causal explanation.

Worked solution and explanation

Add the following code below the worked example, or run the matching file in the solutions folder. Compare the result with the business definition before trusting a passing assertion.

redundant = np.column_stack([np.ones(len(x)), x, x])
assert np.linalg.matrix_rank(redundant) < redundant.shape[1]
simple_mse = np.mean((y - simple_design @ simple_coef) ** 2)
adjusted_mse = np.mean((y - adjusted_design @ adjusted_coef) ** 2)
assert adjusted_mse <= simple_mse
print('Training MSE:', round(float(simple_mse), 3), round(float(adjusted_mse), 3))

Check your understanding

Why does a lower regression training error not establish causality?

Answer guide

Training error measures how closely a chosen model fits observed outcomes. Extra variables can improve that fit even when the model is confounded, selected, misspecified, or using future information. Causality concerns how outcomes would change under an intervention, which requires additional design and assumptions beyond predictive fit.

Reference and next step

Download the complete course kit: datasets, all examples, solutions, and capstone.

Return to the full Data Analysis with Python and pandas curriculum.

Course index · Previous · Next