Module 7: Reliable Analysis Workflows · Lecture 27 of 32
Move from a successful notebook to a repeatable command. Separate reading, validation, transformation, and delivery; write a manifest that identifies the exact inputs and outputs behind a report.
What you will learn
- Design a pipeline with explicit inputs and output paths.
- Fail before publishing a report when a contract is broken.
- Record hashes, versions, audit counts, and metric definitions.
Understand the problem
Separate responsibilities
The supplied pipeline.py reads the raw orders and customer dimension, validates and cleans orders, enriches them through a checked join, creates KPI tables, and writes a report bundle. The calculation functions return data; delivery functions write files. That separation makes failures easier to test and avoids hidden dependence on interactive state.
Use a stable interface
From the kit root run python pipeline.py --input data/orders_raw.csv --out outputs/my-report. The customer dimension defaults to data/customers.csv and can be supplied with –customers. Explicit paths make the command suitable for CI or a scheduler. A nonzero exit status indicates failure; do not swallow exceptions and leave an apparently successful job with stale outputs.
Write evidence with the result
The bundle includes channel and monthly summaries, an audit JSON, a manifest, and an HTML report. The manifest records input hashes, runtime versions, reporting timezone, grain, and the metric contract. A reviewer should be able to trace a displayed total back to source bytes and code, not just a screenshot.
Delivery needs its own reliability model
Each text artifact is written to a temporary sibling and then replaced, reducing the chance of exposing a half-written file. This does not make the entire multi-file bundle transactional. A production publisher can write a versioned directory, validate every artifact, and update a single pointer only after success. Keep previous reports until the new bundle is verified.
Run the worked example
Download and extract the course kit linked below. Open a terminal in the extracted folder, complete the setup in Lecture 2, and run python run_course.py --lesson 27 --examples-only. Each script starts from the shared dataset; it does not depend on a notebook cell from another lecture. All retail records are fictional and amounts are integer paise unless a chart explicitly says INR.
from pathlib import Path
import numpy as np
import pandas as pd
from course_data import load_orders, load_raw, load_customers, clean_orders, ROOT, OUT
from pipeline import run_pipeline
destination = OUT / 'pipeline-example'
result = run_pipeline(ROOT / 'data' / 'orders_raw.csv', destination,
ROOT / 'data' / 'customers.csv')
assert result['audit']['accepted_rows'] == 239
assert result['manifest']['net_paise'] == 24112420
for filename in ['audit.json', 'manifest.json', 'channel-kpis.csv', 'monthly-kpis.csv', 'analysis-report.html']:
assert (destination / filename).is_file()
print('Report bundle created in outputs/pipeline-example')
print('Net revenue paise:', result['manifest']['net_paise'])
Verified output
Report bundle created in outputs/pipeline-example
Net revenue paise: 24112420
Walk through the result
The example calls the same function used by the command-line interface, so the teaching example and operational command share one implementation. The returned object supports tests without parsing a chart. File-existence checks verify delivery, while the monetary assertion verifies a known reference for this fixed fixture. In a new real export, replace fixed expected totals with appropriate reconciliation checks and reviewed baselines.
Common mistakes and diagnosis
Do not overwrite yesterday's only report before validation succeeds. Do not log passwords or raw personal data to diagnose a schema issue. Avoid relative input paths whose meaning changes with a scheduler's working directory. A job that exits zero after catching every exception creates false confidence. A hash identifies bytes but does not establish that the source is accurate or authorized.
Practice assignment
Run the pipeline twice into two new output directories. Compare the channel and monthly CSV bytes and the input hashes. Explain which metadata might legitimately vary in a production run and how you would distinguish that from a changed analytical result.
Worked solution and explanation
Add the following code below the worked example, or run the matching file in the solutions folder. Compare the result with the business definition before trusting a passing assertion.
second = OUT / 'pipeline-repeat'
again = run_pipeline(ROOT / 'data' / 'orders_raw.csv', second, ROOT / 'data' / 'customers.csv')
for filename in ['channel-kpis.csv', 'monthly-kpis.csv']:
assert (destination / filename).read_bytes() == (second / filename).read_bytes()
assert result['manifest']['orders_sha256'] == again['manifest']['orders_sha256']
Check your understanding
Why is atomic replacement of each file not the same as a transactional report bundle?
Answer guide
A reader can still observe a mixture of old and new files if the process fails between replacements. A versioned bundle and a final pointer switch make the set of artifacts change together from the reader's perspective. That delivery concern is separate from whether each individual file was written completely.
Reference and next step
Download the complete course kit: datasets, all examples, solutions, and capstone.
Return to the full Data Analysis with Python and pandas curriculum.
