Statistics and Linear Algebra Essentials

Statistics and linear algebra give machine learning its working language. Statistics describes uncertainty in data: where values cluster, how much they vary, how samples differ from populations, and how likely a result is under an assumption. Linear algebra describes structured numeric data: feature vectors, weight vectors, matrices of examples, geometric distance, projections, and gradients. The outcome of this lesson is practical fluency: when you see a model input table, a loss function, or a training update, you should be able to explain what the numbers mean and how they move.

In this course, these ideas are not separate prerequisites to memorize before machine learning begins. They are the mechanism underneath fitting, evaluating, and improving models. A classifier compares vectors. Linear regression multiplies a feature matrix by a weight vector. Standardization uses mean and standard deviation. Gradient-based training follows a direction in parameter space that reduces a loss. If these pieces are opaque, model behavior looks mysterious; if they are clear, errors become diagnosable.

Purpose and Outcome

A dataset can be viewed in two complementary ways. Statistically, each column is a variable with a distribution. It has a center, spread, outliers, missing values, and relationships with other variables. Algebraically, each row is a vector in feature space, and the whole dataset is a matrix. A learning algorithm transforms that matrix into parameters that make predictions.

For a supervised regression task, the common shape is simple. Let X be an n by d matrix: n examples and d features. Let y be an n-element target vector. A linear model stores a d-element weight vector w and predicts Xw. The error vector is Xw - y. A loss function summarizes that error, often by averaging squared errors. Training changes w until the loss is smaller.

Distributions, Variance, and Scale

A distribution describes how often values occur. In machine learning, distributions matter because training data is only a sample from the world the model will face later. A model trained on narrow, clean, old, or biased samples may perform poorly when the deployment distribution changes. Summary statistics are the first inspection tools: mean for center, variance and standard deviation for spread, quantiles for rank-based location, and correlation for linear co-movement.

The mean of values x is their sum divided by the count. Variance is the average squared distance from that mean. Standard deviation is the square root of variance, bringing the unit back to the original scale. Standardization converts a value into a z-score: subtract the mean and divide by the standard deviation. A z-score of 2 means the value is two standard deviations above the mean.

Scale is a design choice, not cosmetic formatting. If one feature is measured in dollars and another in fractions, distance-based models and gradient-based models can be dominated by the large-scale feature. Standardizing numeric features often makes optimization better conditioned and distances more meaningful. Tree models are less sensitive to monotonic scaling, so the trade-off depends on the algorithm.

Vectors, Matrices, and Model Geometry

A vector is an ordered list of numbers. In ML, a feature vector might represent one house as [size, bedrooms, age], one document as word counts, or one user as behavioral measurements. A matrix stacks vectors into rows. Matrix notation lets us write operations for many examples at once instead of looping through one record at a time.

The dot product multiplies corresponding vector entries and sums them. It is the core operation in a linear model: each feature contributes according to its weight. If x = [2, 3] and w = [4, -1], then x dot w = 2*4 + 3*(-1) = 5. Positive weights push the prediction up when the feature grows; negative weights push it down.

Distance measures how far vectors are from each other. Euclidean distance is straight-line distance. Manhattan distance sums absolute coordinate differences. Cosine similarity measures angle rather than magnitude, which is useful when direction matters more than length, such as comparing text embeddings. These choices affect which examples appear similar and therefore change nearest-neighbor models, clustering, retrieval, and anomaly detection.

Gradients and Learning from Error

A gradient is a vector of partial derivatives. Each component says how the loss would change if one parameter increased slightly. If the gradient component is positive, increasing that parameter increases loss, so gradient descent moves it downward. If the component is negative, gradient descent moves it upward. The learning rate controls step size.

For mean squared error in linear regression, the loss is the average of squared residuals. The residual vector is prediction minus target. The gradient with respect to weights is proportional to X transpose times the residuals. This is why matrix shape matters. X has shape n by d, residuals have shape n, and X transpose has shape d by n. The product gives a d-element gradient, one update direction per weight.

API Anatomy in Code

Even without a specialized numeric library, the core anatomy is visible: a sequence of observations, column summaries, vector operations, matrix-vector multiplication, a loss, and a gradient update. Production code usually delegates heavy numeric work to optimized libraries, but the mathematical contract is the same: align shapes, preserve feature order, control scale, and check assumptions before interpreting model output.

Example 1: Mean, Variance, and Z-Scores

This example computes population variance and z-scores for one feature column. The expected output shows values below the mean as negative and values above the mean as positive. A standard deviation of zero is rejected because z-scores would require division by zero.

from math import sqrt

def summarize(values: list[float]) -> tuple[float, float, list[float]]:
    if not values:
        raise ValueError("values must be non-empty")
    mean = sum(values) / len(values)
    variance = sum((value - mean) ** 2 for value in values) / len(values)
    std = sqrt(variance)
    if std == 0:
        raise ValueError("z-scores require non-zero variance")
    z_scores = [(value - mean) / std for value in values]
    return mean, variance, z_scores

mean, variance, z_scores = summarize([2.0, 4.0, 4.0, 10.0])
print(round(mean, 2))
print(round(variance, 2))
print([round(score, 2) for score in z_scores])

The output is 5.0, 9.0, and z-scores [-1.0, -0.33, -0.33, 1.67]. The value 10 is high relative to this tiny distribution, while 2 is one standard deviation below the mean.

Example 2: Dot Product, Norm, and Cosine Similarity

This example compares two vectors. Dot product alone grows with magnitude, while cosine similarity normalizes by vector lengths and focuses on direction. That distinction matters when comparing documents or embeddings where longer vectors should not automatically look more similar.

from math import sqrt

def dot(left: list[float], right: list[float]) -> float:
    if len(left) != len(right) or not left:
        raise ValueError("vectors must have the same non-zero length")
    return sum(a * b for a, b in zip(left, right))

def norm(vector: list[float]) -> float:
    return sqrt(dot(vector, vector))

def cosine_similarity(left: list[float], right: list[float]) -> float:
    denominator = norm(left) * norm(right)
    if denominator == 0:
        raise ValueError("cosine similarity requires non-zero vectors")
    return dot(left, right) / denominator

query = [1.0, 2.0, 0.0]
document = [2.0, 4.0, 1.0]
print(dot(query, document))
print(round(cosine_similarity(query, document), 3))

The dot product is 10.0. The cosine similarity is approximately 0.976, so the vectors point in nearly the same direction even though the second vector has an extra component.

Example 3: One-Feature Gradient Descent

This example fits a line through points where the target is roughly twice the feature. It uses one weight and no intercept to keep the mechanism visible. Each epoch computes predictions, residuals, mean squared error, the gradient, and a new weight.

def train_weight(xs: list[float], ys: list[float], learning_rate: float, epochs: int) -> float:
    if len(xs) != len(ys) or not xs:
        raise ValueError("features and targets must align")
    weight = 0.0
    for _ in range(epochs):
        predictions = [weight * x for x in xs]
        residuals = [prediction - y for prediction, y in zip(predictions, ys)]
        gradient = 2 * sum(x * residual for x, residual in zip(xs, residuals)) / len(xs)
        weight = weight - learning_rate * gradient
    return weight

xs = [1.0, 2.0, 3.0]
ys = [2.0, 4.0, 6.0]
weight = train_weight(xs, ys, learning_rate=0.05, epochs=25)
print(round(weight, 3))
print(round(weight * 4.0, 3))

The learned weight is close to 2.0, and the prediction for input 4 is close to 8.0. More epochs move the weight nearer to the exact solution, while a learning rate that is too large can make training oscillate or diverge.

Design Choices and Trade-Offs

Population variance divides by n; sample variance divides by n - 1. Use population variance when the values are the complete group you intend to describe, and sample variance when estimating spread in a larger population from a sample. Many preprocessing pipelines care more about consistent scaling than unbiased statistical estimation, but the distinction should still be intentional.

Feature scaling improves many algorithms, but scaling must be fitted only on training data and then applied to validation, test, and production data. Fitting a scaler on all data leaks information from evaluation examples into training. That can make validation look better than real deployment.

Matrix operations are concise and fast when shapes are correct, but shape mistakes can silently produce wrong results in permissive libraries through broadcasting. Defensive checks are useful near data boundaries: row counts must match targets, feature order must be stable, missing values must be handled consistently, and categorical encodings must keep the same columns between training and inference.

Failure Modes and Troubleshooting

Symptom: training loss becomes nan or explodes. Cause: the learning rate may be too high, features may be on very different scales, or input values may include infinities. Diagnose: print the first few feature summaries, check min and max values, and log loss after each epoch. Correct: lower the learning rate, standardize features, clip impossible values only with a documented reason, and reject non-finite inputs.

Symptom: a nearest-neighbor or clustering model groups records by income while ignoring all other columns. Cause: Euclidean distance is dominated by the largest-scale feature. Diagnose: compare feature standard deviations and recompute distances after standardization. Correct: scale numeric features or choose a metric that matches the problem.

Symptom: validation performance is excellent but production performance is poor. Cause: leakage, train-serving skew, or distribution shift. Diagnose: verify that preprocessing was fitted only on training data, compare feature distributions between training and live data, and check whether future information was present during training. Correct: rebuild the pipeline with separate fit and transform phases, remove leaked features, and monitor live distribution summaries.

Reliability and Performance Implications

Numerical reliability affects model reliability. Large values can overflow squared losses. Tiny differences can disappear with limited precision. Repeated matrix operations can be expensive, so vectorized libraries and appropriate data types matter at scale. Reliability also depends on reproducibility: keep feature order explicit, store preprocessing parameters with the model, and version the training data or extraction query used to produce them.

Security is relevant when numeric data comes from users or external systems. A malformed file with huge dimensions can exhaust memory. Unexpected strings in numeric columns can cause fallback behavior that hides data quality problems. Validate dimensions, types, missingness, and finite numeric ranges before training or inference.

Hands-On Lab: Inspect, Scale, and Fit

Prerequisites: Python 3 and a terminal. No third-party packages are required. The lab creates a tiny in-memory dataset, computes summary statistics, standardizes one feature, trains a one-weight model, verifies the prediction, and then exits without writing files.

1. Create a new scratch Python file or run the following block in a Python REPL.
2. Read the printed mean and standard deviation to confirm the feature scale.
3. Check that the standardized values have a mean close to zero.
4. Train the weight and verify that the prediction for 4 is close to 8.
5. Cleanup: close the REPL or delete the scratch file if you created one.

from math import sqrt

def mean(values: list[float]) -> float:
    return sum(values) / len(values)

def standardize(values: list[float]) -> list[float]:
    center = mean(values)
    variance = mean([(value - center) ** 2 for value in values])
    std = sqrt(variance)
    if std == 0:
        raise ValueError("cannot standardize a constant feature")
    return [(value - center) / std for value in values]

def train_weight(xs: list[float], ys: list[float]) -> float:
    weight = 0.0
    for _ in range(40):
        residuals = [(weight * x) - y for x, y in zip(xs, ys)]
        gradient = 2 * sum(x * residual for x, residual in zip(xs, residuals)) / len(xs)
        weight -= 0.05 * gradient
    return weight

xs = [1.0, 2.0, 3.0]
ys = [2.0, 4.0, 6.0]
scaled = standardize(xs)
weight = train_weight(xs, ys)
print(round(mean(scaled), 6))
print(round(weight, 3))
print(round(weight * 4.0, 3))

Verification succeeds when the standardized mean prints 0.0, the weight is close to 2.0, and the prediction is close to 8.0. If the weight is far away, inspect the learning rate and gradient formula first.

Assessment Exercises

• A feature has values [10, 10, 10]. What fails if you compute z-scores, and what should a preprocessing pipeline do with that feature?
• Two vectors have a high dot product but moderate cosine similarity. Explain what that says about magnitude and direction.
• In a linear regression model, why does the gradient have the same length as the weight vector rather than the same length as the training set?
• You standardize using the full dataset before splitting into train and test sets. Describe the leakage and how to fix the workflow.
• A model trained with raw features converges slowly, but the same model converges quickly after scaling. Explain the role of feature scale in gradient descent.

Summary

Statistics tells you how data varies; linear algebra tells you how models represent and transform that data. Means, variances, standard deviations, vectors, matrices, dot products, distances, and gradients are not abstract extras. They are the working parts behind preprocessing, prediction, similarity, loss, and learning. Use them to inspect data before modeling, reason about algorithm behavior, diagnose failures, and choose trade-offs deliberately.