A-B Tests and Online Experimentation

A-B tests let an MLOps team compare a candidate model with the current production model while real users continue to receive a normal service. The outcome is not merely a chart with two bars. A well-run experiment answers a release question: should this model receive more traffic, stay limited, or be rolled back because its online behavior is worse than the incumbent?

In safe model delivery, online experimentation sits after offline evaluation and before broad promotion. Offline metrics tell you whether a candidate is worth exposing. An A-B test tells you how the candidate behaves inside the production decision loop, where ranking position, latency, feature freshness, user mix, and feedback effects can change the result. The release artifact should include the model version, experiment key, allocation rule, metric definitions, guardrail thresholds, and rollback owner.

Experiment Mechanism

An online experiment has four moving parts: assignment, exposure, measurement, and decision. Assignment maps an experimental unit, usually a user, account, device, query, or session, to a variant. Exposure records that the unit actually had an opportunity to be affected by the variant. Measurement joins exposures to outcomes such as click, conversion, retention, fraud review, latency, or human escalation. The decision rule compares variants against a predeclared success metric and guardrails.

The assignment function must be deterministic. If user u-100 is assigned to model B today, the same experiment key should route that user to B tomorrow unless the allocation changes deliberately. Most platforms hash the experiment key and unit id into a bucket number, then map bucket ranges to variants. This prevents request-by-request flipping, allows reproducible analysis, and makes rollback simple because the service can disable the experiment and route every request to the incumbent.

Exposure is different from assignment. A user can be assigned to B but never reach the ranking surface because they did not open the app, failed authentication, or hit a cached response. Counting assigned-but-unexposed users dilutes treatment effects. Counting outcomes without an exposure record creates attribution errors. Production systems usually emit a compact exposure event containing experiment key, variant, unit id or privacy-preserving surrogate, timestamp, model version, request id, and surface.

The analysis layer then estimates treatment effect. For a binary conversion metric, this may be the treatment conversion rate minus the control conversion rate. For revenue or latency, it may be a mean, percentile, trimmed mean, or ratio metric. In model delivery, guardrails matter as much as the main metric: a recommender that raises clicks but increases p95 latency, complaint rate, or policy violations is not a successful release.

API Anatomy

Experiment configuration should make the operational contract explicit. A typical configuration has an experiment key, status, owner, hypothesis, unit type, variants, traffic allocation, model artifact identifiers, exposure event name, primary metric, guardrail metrics, start criteria, stop criteria, and rollback behavior. The serving path needs a small API: assign a variant, load the matching model or configuration, emit exposure, score the request, emit outcome-linked telemetry, and fall back to the control model when any required dependency is unavailable.

Terminology is important. Control is the current behavior or accepted baseline. Treatment is the candidate behavior. Randomization unit is what receives one stable assignment. Traffic allocation is the percentage eligible for the experiment and the split among variants. Ramp is an intentional increase in allocation after health checks pass. Sample ratio mismatch, often shortened to SRM, means the observed split differs enough from the configured split that assignment or logging may be broken.

Example 1: Stable Assignment

The first example implements hash-based assignment. The exact users assigned to A or B depend on the hash value, but the behavior is deterministic: the same user and experiment key always produce the same variant. Changing the experiment key creates a fresh randomization, which is useful when a new experiment must not inherit a previous split.

from hashlib import sha256


def assign_variant(user_id: str, experiment_key: str, traffic_percent: int = 100) -> str:
    if not 0 <= traffic_percent <= 100:
        raise ValueError("traffic_percent must be between 0 and 100")
    digest = sha256(f"{experiment_key}:{user_id}".encode("utf-8")).hexdigest()
    bucket = int(digest[:8], 16) % 100
    if bucket >= traffic_percent:
        return "holdout"
    return "B" if bucket % 2 else "A"

for uid in ["u-100", "u-104", "u-109", "u-110"]:
    print(uid, assign_variant(uid, "ranker-v7"))

Running the block prints u-100 A, u-104 B, u-109 A, and u-110 B. The important property is not which side a specific user lands on; it is that repeated calls are stable and that setting traffic_percent below 100 returns holdout for users outside the configured bucket range.

Example 2: Metric Calculation

The next example uses tiny conversion arrays to show metric anatomy. A production analysis would include confidence intervals, covariate adjustment, filtering rules, and minimum sample sizes, but the core calculation is still a difference between treatment and control outcomes defined before the experiment starts.

from statistics import mean

control = [0, 1, 0, 0, 1, 0, 1, 0]
treatment = [1, 1, 0, 1, 1, 0, 1, 0]

control_rate = mean(control)
treatment_rate = mean(treatment)
absolute_lift = treatment_rate - control_rate
relative_lift = absolute_lift / control_rate

print(f"control={control_rate:.3f}")
print(f"treatment={treatment_rate:.3f}")
print(f"absolute_lift={absolute_lift:.3f}")
print(f"relative_lift={relative_lift:.1%}")

The deterministic output is control=0.375, treatment=0.625, absolute_lift=0.250, and relative_lift=66.7%. That does not automatically justify launch. The metric has only eight observations per group and no guardrails. In a real release review, this result would be treated as a calculation example, not as sufficient evidence.

Example 3: Detecting Assignment Breakage

SRM is one of the fastest ways to catch a broken experiment. If a 50/50 split produces 6,200 users in A and 3,800 in B, the model result is not trustworthy until assignment and event collection are investigated. The following check uses the chi-square statistic for a two-cell split and a conservative threshold.

def check_sample_ratio(observed_a: int, observed_b: int, expected_b_share: float = 0.5) -> str:
    total = observed_a + observed_b
    if total <= 0:
        raise ValueError("at least one assignment is required")
    expected_b = total * expected_b_share
    expected_a = total - expected_b
    chi_square = ((observed_a - expected_a) ** 2 / expected_a) + ((observed_b - expected_b) ** 2 / expected_b)
    return "investigate" if chi_square > 10.83 else "assignment looks plausible"

print(check_sample_ratio(4980, 5020))
print(check_sample_ratio(6200, 3800))

The expected output is assignment looks plausible for 4,980 versus 5,020 and investigate for 6,200 versus 3,800. This check does not prove a correct experiment, but it can stop a bad readout before a model is promoted from corrupted evidence.

Design Choices

The randomization unit is the first major design choice. User-level assignment is common for personalization models because it avoids showing a person inconsistent recommendations. Query-level assignment can be better for search relevance when the same user issues many unrelated queries. Account-level assignment is often required in B2B products because multiple users share workflows and outcomes. The wrong unit can create interference: one user sees treatment output while another user in the same account changes behavior because of it.

The control must be a real baseline. Sometimes control is the previous model artifact with the same feature pipeline. Sometimes it is a rule-based fallback. If the treatment changes both model and feature generation, the experiment cannot isolate which change caused the result. That may be acceptable for a product release, but it should be named honestly as a system experiment rather than a model-only experiment.

Allocation also has trade-offs. A 1% canary catches catastrophic latency or error problems with limited blast radius, but it rarely has enough power to detect business metric changes. A 50/50 split is statistically efficient, but it exposes more users to a candidate that may be worse. Sequential ramps combine both ideas: start small, check service guardrails and SRM, then increase allocation after the predeclared checks pass.

Failure Modes and Troubleshooting

Symptom: the dashboard shows a strong treatment win, but the observed split is far from the configured allocation. Likely cause: SRM from a buggy hash key, missing exposure events, CDN caching, bot filtering, or eligibility logic applied differently by variant. Diagnostics: compare assignment logs with exposure logs by variant, check the hash input, segment by platform and geography, and verify that both variants emit the same event schema. Correction: pause the experiment, fix assignment or logging, restart with a new experiment key, and discard the contaminated readout.

Symptom: offline AUC improves, but online conversion falls. Likely cause: the offline label or evaluation slice does not match the online objective, or the model changes ranking diversity, calibration, or latency in a way the offline metric missed. Diagnostics: inspect per-segment metrics, feature freshness, score distributions, ranking positions, and latency percentiles. Correction: tighten offline gates, add guardrails for the harmed behavior, and rerun with a narrower eligible population.

Symptom: users alternate between old and new recommendations. Likely cause: assignment used request id or session id when the product required user-level stability. Diagnostics: replay several requests for the same user and compare assigned variants. Correction: change the randomization unit to user or account, invalidate inappropriate caches, and restart the experiment because previous exposure is inconsistent.

Reliability, Performance, and Privacy

The serving path should treat experimentation as a low-latency decision, not as a remote analytics query. Assignment can run in process with a local configuration snapshot. Model loading should be resolved before request handling, and fallback should route to the control model when a treatment artifact is missing. Exposure emission should be buffered or asynchronous, but the system must monitor dropped events because missing exposure data invalidates analysis.

Experiment data can be sensitive because it links behavior, outcomes, and model decisions. Store only the identifiers needed for analysis, apply retention limits, and avoid logging raw features that expose private attributes. For regulated or high-impact model use cases, experiment review should include eligibility criteria, harm metrics, and a stop rule that does not depend on waiting for the final business metric.

Hands-On Lab: Local Experiment Harness

Prerequisites: Python 3, a shell, and permission to create a temporary directory. No external service is required. Steps: create a scratch directory, place the three code examples into separate files, and run each with Python. Next, call the assignment function twice for the same user and confirm the variant does not change. Change the experiment key from ranker-v7 to ranker-v8 and confirm that at least some assignments can change. Then run the metric example and record the absolute lift. Finally, run the SRM example and confirm that the imbalanced split is flagged for investigation.

Verification: the assignment script should print four stable variants on repeated runs; the metric script should print conversion rates of 0.375 and 0.625; the SRM script should print one plausible result and one investigation result. Cleanup: delete the scratch directory. If you adapted the harness inside a repository, remove generated logs and do not commit synthetic experiment output unless it is part of a test fixture.

Assessment Exercises

1. A recommendation experiment uses session-level assignment and support tickets report inconsistent recommendations. Explain the mechanism causing the inconsistency and choose a better randomization unit.
2. A treatment raises click-through rate by 2% but increases p95 latency by 180 milliseconds. Define the launch decision you would make and the additional evidence you would request.
3. An experiment configured as 90/10 reports 70/30 exposures. List two possible causes, one diagnostic query or log comparison, and the correction path.
4. A fraud model treatment reduces manual reviews but increases missed fraud in one geography. Explain why an aggregate win can still be unsafe for promotion.
5. Design a rollback rule for a model experiment whose primary metric takes seven days to mature but whose error rate and latency are available immediately.

Summary

A-B testing for MLOps is a controlled routing and measurement system for model releases. The mechanism depends on stable assignment, precise exposure logging, predeclared metrics, guardrails, and an explicit decision rule. Good experiments separate control from treatment, choose the right randomization unit, detect SRM early, and protect users with ramps and rollback. The result is not certainty; it is disciplined evidence for deciding whether a candidate model should receive more production traffic.