Anomaly Detection

Anomaly detection finds observations that do not fit the pattern learned from mostly normal data. In this lesson the outcome is practical: given measurements such as transaction amounts, server metrics, sensor readings, or customer behavior features, you will build a detector that assigns an anomaly score, choose a threshold, and explain why a point was flagged.

This belongs in the Unsupervised Learning section because the usual starting point is a data set without reliable anomaly labels. The model is not learning a named class in the same way a classifier does. It is learning a description of common structure and then asking which new observations are far from, sparse within, or difficult to isolate from that structure.

What an Anomaly Detector Learns

An anomaly is unusual relative to a reference population. A payment of 900 dollars may be normal for wholesale customers and strange for a student meal card. A CPU spike may be normal during a batch job and suspicious at idle time. The first design choice is therefore the comparison group: one global detector, one detector per entity, or a detector conditioned on time, segment, or season.

Most algorithms return a continuous score before they return a yes or no decision. Higher scores may mean more abnormal, as with reconstruction error, while lower scores may mean less normal, as with density estimates. A threshold turns the score into a flag. If labels are unavailable, the threshold often comes from a contamination assumption, such as flagging the highest one percent of scores. That assumption is operational, not magical: if the true rate is much lower, analysts will see many false alarms; if it is higher, important cases may be missed.

Internal Mechanisms

Several families of detectors are common. Distance based detectors compare a point with nearby points. A point far from its neighbors receives a high anomaly score. Density based detectors estimate whether a point lies in a sparse region. Reconstruction based detectors learn to compress and rebuild normal observations; large reconstruction error suggests the point does not follow the learned structure. Isolation based detectors, such as isolation forests, repeatedly split feature space. Unusual points tend to be separated by fewer random splits because they sit alone or at the edge of the distribution.

Isolation Forest is a useful default for tabular numeric data. Each tree picks a random feature and a random split value between that feature’s observed minimum and maximum. The path length is the number of splits needed to isolate a sample in a leaf. Normal samples, surrounded by many similar samples, usually require longer paths. Anomalies often require shorter paths. The forest averages path lengths across trees and converts them to a score. Important configuration terms include n_estimators for the number of trees, max_samples for the number of rows sampled per tree, contamination for the expected flagged fraction when a decision threshold is needed, and random_state for reproducibility.

API Anatomy

In scikit-learn, anomaly detectors follow the estimator pattern. You call fit(X) on historical reference data, then use score_samples(X), decision_function(X), or predict(X) on new rows. For Isolation Forest, predict returns 1 for inliers and -1 for outliers. The decision_function is shifted so values below zero are treated as outliers. Preprocessing matters because many detectors depend on numeric scale. A pipeline keeps imputation, scaling, and the detector fitted together on the same training data.

Example 1: Z Scores for One Metric

The simplest detector compares each value with the mean and standard deviation of a single reference metric. It is transparent and fast, but it assumes the reference distribution is roughly stable and not dominated by extreme values.

from statistics import mean, pstdev

latency_ms = [98, 102, 101, 99, 97, 103, 100, 420]
avg = mean(latency_ms)
sigma = pstdev(latency_ms)

for value in latency_ms:
    z = (value - avg) / sigma if sigma else 0.0
    label = "anomaly" if abs(z) > 2.0 else "normal"
    print(f"{value:3d} ms  z={z:5.2f}  {label}")

The expected behavior is that 420 receives the largest positive z score and is flagged by the threshold, while the values clustered near 100 are normal. This approach is easy to explain to an operator: the alert is about distance from the recent baseline. It performs poorly when the baseline has trends, multiple modes, or heavy tails.

Example 2: Isolation Forest on Two Features

Now use two features: request count and error rate. The unusual row has both high volume and high error rate, a combination that is easier to detect jointly than by either feature alone.

from sklearn.ensemble import IsolationForest

X = [
    [100, 0.01], [105, 0.02], [98, 0.01], [110, 0.02],
    [95, 0.00], [102, 0.01], [108, 0.02], [350, 0.22],
]

model = IsolationForest(contamination=0.125, random_state=7)
model.fit(X)

for row, prediction, score in zip(X, model.predict(X), model.decision_function(X)):
    label = "anomaly" if prediction == -1 else "normal"
    print(f"requests={row[0]:3.0f} error_rate={row[1]:.2f} score={score: .3f} {label}")

With a contamination value of 0.125, one of the eight rows is expected to be flagged. The row [350, 0.22] should be the anomaly. The exact score can vary by library implementation details, so downstream code should not depend on a hard-coded decimal value.

Example 3: Pipeline with Scaling and New Data

A realistic workflow fits preprocessing on historical data and then scores future rows. Scaling is included because features such as dollars and rates can live on very different numeric ranges.

from sklearn.ensemble import IsolationForest
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

history = [
    [42.0, 3.0, 0.02], [39.0, 2.0, 0.01], [45.0, 4.0, 0.03],
    [41.0, 3.0, 0.02], [38.0, 2.0, 0.01], [44.0, 3.0, 0.02],
    [43.0, 4.0, 0.03], [40.0, 2.0, 0.01], [46.0, 4.0, 0.03],
]
new_rows = [[43.0, 3.0, 0.02], [120.0, 18.0, 0.30]]

pipeline = make_pipeline(
    StandardScaler(),
    IsolationForest(contamination=0.10, random_state=11),
)
pipeline.fit(history)

for row, prediction in zip(new_rows, pipeline.predict(new_rows)):
    print(row, "anomaly" if prediction == -1 else "normal")

The first new row resembles the reference history and should be normal. The second combines a much larger amount, more events, and a higher rate, so it should be flagged. The main lesson is procedural: fit transformers only on reference data, then reuse the fitted pipeline for new observations.

Design Choices and Trade-offs

The detector’s training window controls what normal means. A short window adapts quickly but may forget rare normal events. A long window is stable but may hide recent changes. Per-user or per-device models can catch personal deviations, but they need enough history for each entity. A global model works with less data per entity but may unfairly compare small and large populations.

Thresholds should be chosen for the cost of action. Blocking a transaction requires a much stricter threshold than adding it to a review queue. In many systems the right output is not automatic rejection; it is a ranked list with evidence, such as the features that were farthest from baseline and nearby normal examples for comparison.

Failure Modes and Troubleshooting

If nearly everything is flagged after deployment, first check feature order, units, and preprocessing. A common cause is fitting on dollars but scoring cents, or fitting a scaler during training and forgetting to apply the same scaler at inference. Diagnose by printing feature summaries before and after preprocessing for both training and live batches. Correct by packaging preprocessing and the estimator in one pipeline and adding a schema test for column names and ranges.

If nothing is ever flagged, the threshold may be too loose or the training set may contain the same anomalies you want to detect. Inspect the score distribution, review the top scored rows manually, and compare training and recent data. Correct by cleaning the reference window, lowering the threshold, or training separate detectors for different contexts.

If alerts spike every Monday morning, the model is probably treating seasonality as abnormal. Plot scores by hour and day. If the pattern is expected, add time context, train separate baselines for recurring periods, or use features that compare a row with the same period in previous cycles.

Security, Performance, and Reliability

Anomaly scores can influence sensitive decisions, so store enough explanation for audit without storing unnecessary raw personal data. Treat model files and threshold configuration as change-controlled artifacts because a threshold change can alter who is investigated or blocked. For performance, batch scoring is usually cheaper than row-at-a-time scoring, and tree count controls the latency versus stability trade-off. For reliability, monitor the percentage of rows flagged, the score distribution, missing feature rates, and drift in important feature summaries.

Hands-on Lab

Prerequisites: Python with scikit-learn installed, a terminal, and permission to create a temporary script. The lab builds a detector for synthetic account activity.

  1. Create a file named lab_anomaly.py with the code below.
  2. Run python lab_anomaly.py.
  3. Verify that the row with a much larger amount and count is printed as an anomaly.
  4. Change contamination from 0.10 to 0.30 and run again. More rows should be flagged because the threshold assumes more anomalies.
  5. Cleanup by deleting the temporary script after the experiment.
from sklearn.ensemble import IsolationForest
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rows = [
    [20, 1], [22, 1], [19, 2], [24, 1], [21, 2],
    [23, 1], [18, 1], [25, 2], [200, 12], [20, 2],
]

model = make_pipeline(
    StandardScaler(),
    IsolationForest(contamination=0.10, random_state=21),
)
model.fit(rows)

predictions = model.predict(rows)
for row, prediction in zip(rows, predictions):
    label = "anomaly" if prediction == -1 else "normal"
    print(f"amount={row[0]:3d} count={row[1]:2d} {label}")

Verification is not just seeing one alert. Confirm that ordinary rows remain normal, then deliberately change the threshold and observe how the review volume changes. That experiment connects model configuration to operational workload.

Assessment Exercises

  • You train a detector on all transactions from the last year and it flags most December purchases. What context or feature change would you investigate first?
  • Explain why predict is convenient for automation but decision_function is better for a review queue.
  • A fraud team can review only 200 cases per day. Describe how you would set and evaluate the anomaly threshold without reliable labels.
  • Given a detector that works globally but misses unusual behavior within one large customer account, argue for or against per-account baselines.
  • Write a test that would catch swapped feature columns before scoring live data.

Summary

Anomaly detection learns a model of ordinary structure and uses scores to rank observations by how poorly they fit that structure. The hardest parts are defining the comparison group, preserving the exact preprocessing used during fitting, choosing a threshold that matches the cost of action, and investigating failures through score and feature distributions. In unsupervised learning, the model can surface useful surprises, but humans still own the definition of unusual and the consequences of acting on it.