Course index · Previous · Next

Sampling, Selection Bias, and the Unit of Analysis

Module 6: Statistics and Sound Decisions · Lecture 21 of 32

A large dataset can still be systematically biased. Use a controlled simulation to separate random sampling variability from selection bias, and choose whether orders or customers are the independent observations.

What you will learn

• Distinguish a target population from an observed sample.
• Demonstrate selection bias that more rows cannot fix.
• Identify repeated observations and clustering.

Understand the problem

Name the population you want to describe

A store export contains recorded orders during a period, not every person who considered buying. A survey of purchasers excludes people who abandoned checkout. A dashboard of identified customers excludes anonymous buyers. Before calculating uncertainty, describe how an observation enters the dataset and who is missing from that process.

Random variation versus systematic selection

A random sample can vary around a population value. A sample selected only from high spenders targets a different distribution, even if it contains thousands of rows. Increasing its size makes the biased estimate more precise about the wrong population. A confidence interval addresses a sampling model; it does not repair a selection mechanism.

Choose the independent unit

Repeated orders from one customer can share preferences, region, and marketing exposure. Treating every order as an independent customer overstates the effective information for customer-level questions. Aggregate to customer level or use a clustered method appropriate to the design. The unit of analysis should match the decision and the sampling process.

Use simulation to understand, not to manufacture evidence

The following population is invented so its mean is known. A fixed random generator seed makes the demonstration repeatable in the pinned environment. The simulation proves how the chosen sampling mechanisms behave under the specified construction. It says nothing directly about a real retailer's average spend or customer distribution.

Run the worked example

Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 21 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.

from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT

rng = np.random.default_rng(20261024)
population = rng.lognormal(mean=6.0, sigma=0.8, size=100000)
random_sample = rng.choice(population, size=2000, replace=False)
selected = population[population >= np.quantile(population, 0.8)]
biased_sample = rng.choice(selected, size=2000, replace=False)
truth = population.mean()
print('Simulated population mean:', round(float(truth), 2))
print('Random sample mean:', round(float(random_sample.mean()), 2))
print('Selected high-spender mean:', round(float(biased_sample.mean()), 2))
assert biased_sample.mean() > truth
assert len(random_sample) == len(biased_sample)

Verified output

Simulated population mean: 552.4
Random sample mean: 547.15
Selected high-spender mean: 1342.18

Walk through the result

The two samples have the same number of observations, yet only one samples the full simulated population. The high-spender sample is restricted by construction, so its elevated mean is not a random surprise. The fixed seed stabilizes the demonstration. Exact random samples can change across algorithms or library versions, which is one reason the kit records the environment rather than promising universal byte-identical randomness.

Common mistakes and diagnosis

Do not equate dataset size with representativeness. Do not use a narrow confidence interval to claim a selected sample represents everyone. Weighting can help only when its assumptions and inclusion probabilities are defensible. An order-level sample and a customer-level sample answer different questions. Do not generalize the synthetic retail data to actual market behavior.

Practice assignment

Repeat random sampling twenty times and record the sample means. Compare their spread with the high-spender sample mean. Then aggregate the retail table to one row per identified customer and count how many independent customer units that table contains.

means = np.array([rng.choice(population, size=2000, replace=False).mean() for _ in range(20)])
assert means.mean() < biased_sample.mean()
orders = load_orders()
customer_units = orders.dropna(subset=['customer_id']).groupby('customer_id')['net_paise'].sum()
print('Identified customer units:', len(customer_units))
assert customer_units.index.is_unique

Check your understanding

Can a larger biased sample be less useful than a smaller random sample?

Reference and next step

• NumPy random sampling

Download the complete course kit: datasets, all examples, solutions, and capstone.

Return to the full Data Analysis with Python and pandas curriculum.

Course index · Previous · Next