Explore Distributions and Investigate Outliers

Module 5: Exploration and Visualization · Lecture 18 of 32

A mean hides the shape of a population. Compare means, medians, quantiles, and unusual observations, then decide whether an extreme value is an error, a legitimate case, or evidence of a different process.

What you will learn

Summarize distributions with robust and non-robust measures.
Use an IQR rule as a review flag rather than an automatic deletion.
Connect unusual values back to source order IDs.

Understand the problem

Look at shape before ranking

Order revenue can be skewed, with many modest purchases and a few large ones. The mean responds strongly to large values; the median describes the middle observation. Quantiles reveal more of the distribution. Include population size and the treatment of returns and cancellations so the summary can be interpreted correctly.

A flag is not a verdict

The interquartile range is the difference between the 75th and 25th percentiles. A common exploratory rule flags values below Q1 minus 1.5 IQR or above Q3 plus 1.5 IQR. The multiplier is a convention, not a universal boundary between valid and invalid data. A legitimate bulk purchase can exceed it. Inspect the source key, quantity, price, discount, and refund before deciding what to do.

Match the statistic to the population

For this lesson use non-cancelled orders, including fully refunded returns. A zero net amount is therefore meaningful, not necessarily missing. If you instead study gross basket size, use gross_paise and say so. Changing the measure or population to obtain a more attractive chart undermines the analysis.

Quantify sensitivity

Compare a result with and without flagged observations as a sensitivity check, but do not silently publish the filtered version. If a few records dominate the conclusion, investigate concentration and report the dependence. Synthetic data patterns are useful for practising this workflow but cannot establish what a real store's distribution should look like.

Run the worked example

Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 18 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.

from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT

orders = load_orders()
eligible = orders.loc[orders['status'].ne('cancelled')].copy()
values = eligible['net_paise'].astype(float) / 100
q1, q3 = values.quantile([0.25, 0.75])
iqr = q3 - q1
flags = values.lt(q1 - 1.5 * iqr) | values.gt(q3 + 1.5 * iqr)
print(values.describe(percentiles=[0.25, 0.5, 0.75, 0.9, 0.95]).round(2).to_string())
print('IQR review flags:', int(flags.sum()))
print('Mean INR:', round(float(values.mean()), 2), 'Median INR:', round(float(values.median()), 2))
assert len(values) == len(eligible)
assert values.ge(0).all()
assert values.quantile(0.95) >= values.median()

Verified output

count     224.00
mean     1076.45
std       920.85
min         0.00
25%       358.20
50%       598.00
75%      1598.00
90%      2397.00
95%      3196.00
max      3196.00
IQR review flags: 0
Mean INR: 1076.45 Median INR: 598.0

Walk through the result

describe reports count, mean, standard deviation, selected percentiles, and extremes. The count reflects the chosen eligible population, not the raw file. The IQR mask remains aligned with order rows, allowing a review table with source keys. If no observations are flagged, that does not prove the data is correct: the planted invalid prices and dates were handled by schema checks before this distribution analysis.

Common mistakes and diagnosis

Do not treat zero net revenue as a missing value. Do not remove a high value solely because it changes the mean. A normal-looking histogram does not prove a normal population. Quantiles depend on the sample and interpolation convention, so tiny differences between tools deserve a method check before being called errors. Never infer a source's completeness from a tidy distribution.

Practice assignment

Export flagged order IDs with their units, price, discount, refund, and net amount. Compare the mean of all eligible orders with the mean of unflagged orders. If there are no flags, state that outcome rather than changing the threshold until you find one.

Worked solution and explanation

Add the following code below the worked example, or run the matching file in the solutions folder. Compare the result with the business definition before trusting a passing assertion.

review = eligible.loc[flags, ['order_id', 'units', 'unit_price_paise', 'discount_paise', 'refunded_paise', 'net_paise']]
review.to_csv(OUT / 'outlier-review.csv', index=False)
unflagged_mean = values.loc[~flags].mean()
assert len(review) == int(flags.sum())
print('Unflagged mean INR:', round(float(unflagged_mean), 2))

Check your understanding

Can a record pass every schema rule and still deserve investigation?

Answer guide

Yes. A value can have a valid type, allowed range, unique key, and consistent arithmetic while still being unusual or incorrectly recorded. Schema validation catches declared contract violations; exploratory analysis identifies patterns worth investigating. Neither alone proves accuracy, and a statistical flag is not sufficient evidence for deletion.

Reference and next step

pandas descriptive statistics

Download the complete course kit: datasets, all examples, solutions, and capstone.

Return to the full Data Analysis with Python and pandas curriculum.

Course index · Previous · Next