Module 7: Reliable Analysis Workflows · Lecture 26 of 32
Scale an analysis by loading only necessary columns and combining additive summaries. Learn which operations can be chunked safely and why duplicates, distinct counts, and averages require more thought than a simple loop over file parts.
What you will learn
- Measure DataFrame memory rather than file size alone.
- Read selected columns in chunks and combine exact sums.
- Recognize operations that need cross-chunk state.
Understand the problem
Load less before buying more memory
A compressed file can expand substantially in memory. String objects, indexes, intermediate joins, and copies add cost. Use memory_usage(deep=True) for a practical estimate, select only required columns with usecols, and choose dtypes consistent with the domain. A category can help repeated labels; a smaller integer dtype requires a proven range.
Additive statistics compose
Revenue sums and row counts can be calculated per chunk and added. An overall mean needs the sum and count from every chunk, not the mean of chunk means. A maximum can combine chunk maxima. Exact distinct counts need a set or database-level strategy that tracks identities across chunks; adding per-chunk distinct counts double-counts repeated keys.
Cleaning raw chunks is harder
The same order can appear in two different chunks. Deduplicating each chunk independently does not remove that cross-boundary duplicate. This example chunks an already validated, unique clean export. A production raw-ingestion pipeline needs a global key strategy, source partition contract, or external storage that can resolve keys across the whole population.
Know when to change engines
Repeated large joins, global sorting, and wide intermediate tables may exceed practical in-memory limits even with chunked ingestion. Consider a database or a suitable out-of-core engine when the workload requires it. Measure elapsed time and peak memory on representative data; a tiny teaching dataset cannot substantiate a production speed claim.
Run the worked example
Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 26 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.
from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT
orders = load_orders()
clean_path = OUT / 'clean-for-chunks.csv'
orders.to_csv(clean_path, index=False)
totals = {}
rows = 0
for chunk in pd.read_csv(clean_path, usecols=['channel', 'net_paise'],
dtype={'channel': 'string', 'net_paise': 'Int64'}, chunksize=50):
rows += len(chunk)
for key, value in chunk.groupby('channel', dropna=False)['net_paise'].sum().items():
totals[key] = totals.get(key, 0) + int(value)
expected = {key: int(value) for key, value in orders.groupby('channel')['net_paise'].sum().items()}
assert totals == expected
assert rows == len(orders)
print('Measured DataFrame bytes:', int(orders.memory_usage(deep=True).sum()))
print('Chunked channel totals paise:', dict(sorted(totals.items())))
Verified output
Measured DataFrame bytes: 67361
Chunked channel totals paise: {'email': 8045280, 'organic': 8311030, 'paid': 7596310, 'unknown': 159800}
Walk through the result
The accumulator uses Python integers, avoiding a floating-point merge of large monetary totals. Only channel and net_paise are loaded from the clean file. Each chunk contributes an additive subtotal, and the final dictionary matches the in-memory reference exactly. The row counter independently verifies that the file was fully processed. This demonstrates correctness, not a speed improvement on 239 rows.
Common mistakes and diagnosis
Do not sum per-chunk medians, quantiles, or distinct counts. Do not discard duplicates separately in each raw chunk and assume global uniqueness. Smaller dtypes can overflow if selected without range checks. Chunking input does not help if every chunk is appended to a list and concatenated into the same huge final table. Avoid timing claims without a measured workload.
Practice assignment
Compute overall AOV from chunked net sum and non-cancelled order count. Do not average chunk AOVs. Verify agreement with the in-memory contract and explain the additional state needed for exact distinct-customer counts.
Worked solution and explanation
Add the following code below the worked example, or run the matching file in the solutions folder. Compare the result with the business definition before trusting a passing assertion.
net_total = 0
eligible_count = 0
for chunk in pd.read_csv(clean_path, usecols=['status', 'net_paise'], dtype={'status':'string', 'net_paise':'Int64'}, chunksize=50):
chosen = chunk.loc[chunk['status'].ne('cancelled')]
net_total += int(chosen['net_paise'].sum())
eligible_count += len(chosen)
reference = orders.loc[orders['status'].ne('cancelled')]
assert np.isclose(net_total / eligible_count, reference['net_paise'].sum() / len(reference))
Check your understanding
Why does chunked deduplication fail when repeated keys cross chunk boundaries?
Answer guide
Each chunk sees only its own rows. A key appearing once in each of two chunks looks unique locally but is duplicated globally. The pipeline needs cross-chunk identity state, a source guarantee that partitions keys uniquely, or a storage engine that enforces the global key contract.
Reference and next step
Download the complete course kit: datasets, all examples, solutions, and capstone.
Return to the full Data Analysis with Python and pandas curriculum.
