Logistic Regression

Logistic regression is a supervised learning model for estimating the probability of a discrete class, most commonly a binary outcome such as churn or not churn, fraud or legitimate, admitted or rejected. Despite the word regression, its usual job is classification. It produces a probability, and a separate decision rule converts that probability into a class label.

In this course, logistic regression is the bridge between simple linear models and modern probabilistic classifiers. You still learn one weight per feature, as in linear regression, but the model output is passed through a sigmoid function so it stays between 0 and 1. That makes the model easy to inspect, fast to train, and useful as a baseline before reaching for more flexible algorithms.

Purpose and Outcome

The practical outcome is not merely saying class 0 or class 1. A useful logistic regression workflow estimates probability, evaluates whether those probabilities are well behaved, and chooses a threshold that matches the cost of false positives and false negatives. A medical screening model might use a low threshold to catch more true cases. A model that automatically locks user accounts might use a higher threshold because false accusations are expensive.

A fitted binary logistic regression model has three visible parts: feature preprocessing, a linear score, and a probability transformation. The learned linear score is often called the logit. Positive weights increase the log odds of the positive class when that feature increases, holding other features fixed. Negative weights decrease those log odds. This interpretation is one reason logistic regression remains important even when more complex models have higher raw accuracy.

Mechanism: From Score to Probability

The model begins with a weighted sum:

import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

def predict_probability(features, weights, bias):
    score = bias + sum(x * w for x, w in zip(features, weights))
    return sigmoid(score)

weights = [1.2, -0.8]
bias = -0.3
print(round(predict_probability([2.0, 1.0], weights, bias), 3))
print(round(predict_probability([0.0, 3.0], weights, bias), 3))

The first example shows the internal calculation without any library. For features [2.0, 1.0], the score is -0.3 + 2.0 * 1.2 + 1.0 * -0.8, which is 1.3. The sigmoid of 1.3 is about 0.786, so the model estimates a 78.6 percent probability for the positive class. For [0.0, 3.0], the score is -2.7 and the probability is about 0.063. The expected output is:

0.786
0.063

The sigmoid function is S-shaped. Very large positive scores approach probability 1, very large negative scores approach probability 0, and score 0 maps to probability 0.5. This matters because equal changes in score do not always create equal changes in probability. Moving from score 0 to 1 changes probability substantially; moving from score 8 to 9 barely changes it because the model is already almost certain.

Training chooses weights and bias that minimize log loss, also called binary cross-entropy. For a positive example, log loss heavily penalizes a predicted probability near 0. For a negative example, it heavily penalizes a predicted probability near 1. This objective is smoother and more informative than simply counting wrong labels, which lets optimization algorithms improve the weights gradually.

API Anatomy

Most library APIs expose the same conceptual pieces. You supply a two-dimensional feature matrix X, a one-dimensional target vector y, and optional configuration such as regularization strength, class weighting, and the optimization solver. After fitting, the model exposes learned coefficients, an intercept, predict_proba for probabilities, and predict for thresholded class labels.

Feature scaling is often important. Logistic regression can fit unscaled numeric features, but optimization and regularization are easier to reason about when features share comparable ranges. One feature measured in dollars and another measured as a 0-or-1 flag should not receive different regularization pressure just because of units. Categorical features usually need one-hot encoding. Missing values must be imputed or removed before fitting because the model cannot learn from undefined arithmetic.

Example 2: Fitting Weights

The next example fits a tiny logistic regression model with gradient descent. It uses one feature so the update math is visible. Each pass computes probabilities, compares them with labels, and nudges the weight and bias in the direction that reduces log loss.

import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

x_values = [-2.0, -1.0, 0.0, 1.0, 2.0]
y_values = [0, 0, 0, 1, 1]
weight = 0.0
bias = 0.0
learning_rate = 0.4

for _ in range(300):
    grad_w = 0.0
    grad_b = 0.0
    for x, y in zip(x_values, y_values):
        probability = sigmoid(weight * x + bias)
        error = probability - y
        grad_w += error * x
        grad_b += error
    weight -= learning_rate * grad_w / len(x_values)
    bias -= learning_rate * grad_b / len(x_values)

print(round(weight, 2), round(bias, 2))
for x in [-1.5, 0.5, 1.5]:
    print(x, round(sigmoid(weight * x + bias), 3))

The data says larger x values are more likely to belong to class 1. Starting from zero weight and zero bias, every example initially receives probability 0.5. After training, the weight becomes positive, so larger values produce larger logits and higher probabilities. The exact printed values are deterministic for this loop and rounded output:

3.31 -1.67
-1.5 0.001
0.5 0.496
1.5 0.965

The middle prediction is near 0.5 because the fitted decision boundary is close to 0.5 on the input axis. This is the point where the logit is near zero. In real projects you rarely write gradient descent by hand, but understanding this loop makes the library behavior less mysterious: training is repeated probability estimation followed by loss-driven parameter updates.

Example 3: Thresholds and Metrics

A probability is not yet a decision. The following example applies two thresholds to the same probability list and counts confusion-matrix outcomes.

def confusion_counts(y_true, probabilities, threshold):
    counts = {"tp": 0, "fp": 0, "tn": 0, "fn": 0}
    for actual, probability in zip(y_true, probabilities):
        predicted = 1 if probability >= threshold else 0
        if actual == 1 and predicted == 1:
            counts["tp"] += 1
        elif actual == 0 and predicted == 1:
            counts["fp"] += 1
        elif actual == 0 and predicted == 0:
            counts["tn"] += 1
        else:
            counts["fn"] += 1
    return counts

y_true = [0, 0, 1, 1, 1]
probabilities = [0.10, 0.40, 0.45, 0.70, 0.90]
print(confusion_counts(y_true, probabilities, 0.50))
print(confusion_counts(y_true, probabilities, 0.80))

At threshold 0.50, two positives are caught and one positive is missed. At threshold 0.80, only the strongest positive is caught, so false negatives increase. The expected output is:

{'tp': 2, 'fp': 0, 'tn': 2, 'fn': 1}
{'tp': 1, 'fp': 0, 'tn': 2, 'fn': 2}

The right threshold depends on consequences. If missing a positive case is worse than reviewing an extra case, lower the threshold. If a false alarm is worse than a missed opportunity, raise it. Accuracy hides this trade-off, especially when classes are imbalanced. Precision, recall, ROC curves, precision-recall curves, and cost-weighted evaluation usually give a better picture.

Design Choices and Trade-offs

Logistic regression is linear in the features after preprocessing. That is a limitation and a strength. It cannot automatically discover complex interactions unless you add interaction features, splines, or other transformations. In return, it trains quickly, needs relatively little data compared with high-capacity models, and gives coefficients that can be inspected.

Regularization controls coefficient size. L2 regularization shrinks weights smoothly and is a common default. L1 regularization can drive some weights to zero, which helps feature selection when many features are weak or redundant. Stronger regularization may improve generalization but can underfit real signals. Weaker regularization may fit the training data better but can overreact to noise.

Class imbalance requires care. If only one percent of examples are positive, a model can reach 99 percent accuracy by always predicting negative. Better approaches include using class weights, collecting more positive examples, evaluating precision and recall, and choosing a threshold based on operational cost. Resampling can help, but it should be done inside the training split only to avoid leaking duplicated or synthesized information into validation.

Failure Modes and Troubleshooting

Symptom: training accuracy is high but validation recall is poor. Cause: the threshold is too high, the classes are imbalanced, or the training data does not represent validation cases. Diagnose: inspect the confusion matrix across multiple thresholds and compare feature distributions by split. Correct: tune the threshold on validation data, use class weighting when appropriate, and rebuild splits so they reflect the real prediction setting.

Symptom: coefficients are extremely large or change wildly across runs. Cause: features may be collinear, poorly scaled, or nearly separable. Near separation means a line almost perfectly divides the classes, so the loss keeps rewarding larger weights. Diagnose: check feature scales, correlations, and predicted probabilities near exactly 0 or 1. Correct: standardize features, increase regularization, remove duplicate signals, or gather examples near the boundary.

Symptom: offline validation looks excellent but production decisions fail. Cause: leakage or training-serving skew. A leaked feature contains information that would not be available at prediction time, such as a post-outcome status. Training-serving skew means preprocessing differs between fitting and deployment. Diagnose: audit every feature timestamp and compare transformed feature samples from training and live inference. Correct: remove future-looking fields and package preprocessing with the model in one reproducible pipeline.

Reliability and Performance

Logistic regression is usually cheap at inference time: compute a dot product, add a bias, apply sigmoid. This makes it reliable for high-volume services and batch scoring. The main reliability risks are data quality, schema drift, and uncalibrated decisions. Monitor input ranges, missing-value rates, class prevalence, probability distributions, and business outcomes. If the model drives consequential decisions, store model version, threshold, feature schema version, and enough non-sensitive diagnostic information to explain a prediction later.

Probabilities should be treated carefully. Logistic regression often gives reasonable probabilities when its assumptions are not badly violated, but calibration should still be checked. Calibration asks whether examples assigned probability 0.8 are positive about 80 percent of the time. If not, use calibration plots and consider Platt scaling or isotonic calibration on held-out data.

Hands-on Lab

Prerequisites: Python 3 and the standard library. No external package is required. The lab trains the one-feature model above, evaluates several thresholds, and verifies that the chosen threshold changes false negatives.

  1. Create a working file named logistic_lab.py.
  2. Add the following complete program.
  3. Run python logistic_lab.py.
  4. Verify that probabilities increase as the input value increases.
  5. Compare the confusion counts for thresholds 0.30, 0.50, and 0.70.
  6. Cleanup by deleting the temporary file when finished.
import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

def train(xs, ys, steps=300, learning_rate=0.4):
    weight = 0.0
    bias = 0.0
    for _ in range(steps):
        grad_w = 0.0
        grad_b = 0.0
        for x, y in zip(xs, ys):
            probability = sigmoid(weight * x + bias)
            error = probability - y
            grad_w += error * x
            grad_b += error
        weight -= learning_rate * grad_w / len(xs)
        bias -= learning_rate * grad_b / len(xs)
    return weight, bias

def confusion(ys, ps, threshold):
    result = {"tp": 0, "fp": 0, "tn": 0, "fn": 0}
    for y, p in zip(ys, ps):
        pred = 1 if p >= threshold else 0
        if y == 1 and pred == 1:
            result["tp"] += 1
        elif y == 0 and pred == 1:
            result["fp"] += 1
        elif y == 0 and pred == 0:
            result["tn"] += 1
        else:
            result["fn"] += 1
    return result

xs = [-2.0, -1.0, 0.0, 1.0, 2.0]
ys = [0, 0, 0, 1, 1]
weight, bias = train(xs, ys)
probabilities = [sigmoid(weight * x + bias) for x in xs]
print(round(weight, 2), round(bias, 2))
print([round(p, 3) for p in probabilities])
for threshold in [0.3, 0.5, 0.7]:
    print(threshold, confusion(ys, probabilities, threshold))

Verification should show a positive weight, a negative bias, and probabilities sorted in the same order as the input values. The 0.30 threshold should classify the borderline positive more aggressively than the 0.70 threshold. If the probabilities are not increasing, inspect the labels and make sure higher feature values correspond to positive examples in the training data.

Assessment Exercises

  • A model predicts fraud probability. False positives annoy customers, while false negatives lose money. Describe how you would choose and validate a threshold without using accuracy alone.
  • Suppose a logistic regression coefficient is positive after standardized preprocessing. Explain what that means in terms of log odds and why it does not prove causation.
  • Your validation score collapses after removing a feature named days_since_resolution. Explain why that may be a good sign rather than a bad one.
  • Design a preprocessing pipeline for numeric age, categorical country, and missing income. State where scaling, encoding, and imputation belong.
  • Compare L1 and L2 regularization for a dataset with thousands of sparse text features. Which would you try first, and what behavior would you inspect?

Summary

Logistic regression learns a linear score, converts it to probability with the sigmoid function, and fits parameters by minimizing log loss. Its simplicity is valuable: coefficients are inspectable, inference is fast, and probability thresholds can be matched to real consequences. Its limits are equally important. Linear decision boundaries, leakage, imbalance, poor scaling, and careless threshold choices can make the model misleading. Used well, logistic regression is a strong supervised-learning baseline and a practical classifier for many real systems.