Module 6: Statistics and Sound Decisions · Lecture 23 of 32
Learn a complete two-proportion experiment calculation using separately simulated randomized data. Specify the assignment unit, primary metric, effect size, uncertainty, and stopping rule before interpreting a p-value.
What you will learn
- Calculate absolute and relative conversion lift.
- Compute an approximate interval and two-sided test for a difference in proportions.
- Identify peeking, multiple testing, and non-random assignment risks.
Understand the problem
Design comes before calculation
The retail order export cannot measure conversion because it has no population of exposed non-buyers or randomized assignment. This lecture therefore creates a separate simulated experiment with 2000 independent users per arm. Each user is assigned once and contributes one binary conversion outcome. A real experiment needs a verified assignment mechanism, stable exposure definition, and predeclared analysis window.
Effect sizes answer different questions
If treatment conversion is 12% and control is 10%, the absolute difference is two percentage points and the relative increase is 20%. Report both the scale and denominators. Relative lift alone can make a small absolute change sound dramatic. A business decision also needs guardrails such as error rate, retention, or revenue, with their own collection contracts.
Interval and test
The approximate interval below uses the unpooled standard error for the difference. The null test uses a pooled rate under equal conversion probabilities. These normal approximations require sufficiently large success and failure counts and independent randomized units. Small counts or more complex designs need methods suited to those conditions.
Commit to the decision rule
Repeatedly checking an ordinary fixed-horizon p-value and stopping when it crosses a threshold changes its error properties. Multiple variants and many metrics create additional opportunities for chance findings. Set the primary metric, minimum detectable effect, duration or sample size, exclusion rules, and analysis plan in advance, or use a valid sequential design.
Run the worked example
Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 23 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.
from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT
from statistics import NormalDist
rng = np.random.default_rng(23)
control = rng.binomial(1, 0.10, size=2000)
treatment = rng.binomial(1, 0.12, size=2000)
pc, pt = control.mean(), treatment.mean()
nc, nt = len(control), len(treatment)
effect = pt - pc
se = np.sqrt(pc * (1-pc)/nc + pt * (1-pt)/nt)
pooled = (control.sum() + treatment.sum()) / (nc + nt)
null_se = np.sqrt(pooled * (1-pooled) * (1/nc + 1/nt))
z = effect / null_se
p_value = 2 * (1 - NormalDist().cdf(abs(float(z))))
print('Control / treatment rates:', round(float(pc), 4), round(float(pt), 4))
print('Absolute lift (percentage points):', round(float(effect * 100), 3))
print('Approximate 95% interval (points):', [round(float(x * 100), 3) for x in [effect-1.96*se, effect+1.96*se]])
print('Two-sided approximate p-value:', round(float(p_value), 5))
assert 0 <= p_value <= 1
assert min(control.sum(), nc-control.sum(), treatment.sum(), nt-treatment.sum()) >= 10
Verified output
Control / treatment rates: 0.0935 0.1085
Absolute lift (percentage points): 1.5
Approximate 95% interval (points): [-0.367, 3.367]
Two-sided approximate p-value: 0.11545
Walk through the result
The simulated generating probabilities differ by two percentage points, but the observed effect need not equal that exact value. The interval and p-value describe the realized randomized sample under their assumptions. A p-value is not the probability that treatment has no effect, nor the probability that the decision is wrong. Practical value depends on effect size, costs, guardrails, and uncertainty, not only a threshold.
Common mistakes and diagnosis
Do not infer conversion rates from purchasers alone. Do not randomize by customer and then analyze correlated sessions as independent units. Do not turn a non-significant result into proof of no difference. A subgroup found after searching many slices is exploratory until confirmed. Never promote a simulated effect to a real site recommendation.
Practice assignment
Calculate relative lift and show how it differs from percentage-point lift. Write a short analysis plan naming one primary metric, one guardrail, the assignment unit, and a fixed stopping rule. State why changing the rule after inspecting results is problematic.
Worked solution and explanation
Add the following code below the worked example, or run the matching file in the solutions folder. Compare the result with the business definition before trusting a passing assertion.
relative_lift = effect / pc
print('Relative lift percent:', round(float(relative_lift * 100), 3))
assert np.isclose((pt / pc - 1), relative_lift)
assert nc == nt == 2000
Check your understanding
What does a p-value measure in this calculation?
Answer guide
Under the equal-rate null model and the stated approximation, it measures how surprising a test statistic at least as extreme as the observed one would be. It does not assign a probability to the null itself. Its interpretation depends on the experiment design, independence, stopping rule, and the set of comparisons examined.
Reference and next step
Download the complete course kit: datasets, all examples, solutions, and capstone.
Return to the full Data Analysis with Python and pandas curriculum.
