Design a Reproducible Data Engineering Project

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: make the same project rebuild the same result

A reproducible data engineering project is one that another engineer, scheduler, or future version of you can run from a clean checkout and get the same meaningful outputs from the same declared inputs. In data engineering, that matters because the product is not only source code. The product also includes datasets, schemas, dependencies, configuration, runtime parameters, and generated artifacts. A notebook that worked on one laptop last month is not reproducible if its input file was overwritten, a package upgraded silently, or a local environment variable changed the branch of logic.

The outcome of this lesson is a concrete design for a small project that can be rebuilt, inspected, and debugged. The pattern is intentionally simple: separate raw and processed data, pin the execution environment, treat configuration as explicit input, make transforms deterministic and idempotent, and write a manifest that records exactly which bytes produced each artifact. Those same ideas scale to Airflow DAGs, dbt projects, Spark jobs, and streaming backfills.

How reproducibility works internally

Reproducibility is a property of the whole dependency graph. A pipeline output is a function of source code, input data, runtime parameters, dependency versions, system services, and time-sensitive behavior. If any of those inputs are implicit, the output cannot be explained from the repository alone. A reproducible project turns those hidden inputs into named, versioned, and testable project assets.

The first internal boundary is the data zone. Raw data is an immutable copy of what arrived from a source system. Processed data is derived and may be deleted and rebuilt. Artifacts are metadata or deliverables such as manifests, validation reports, model features, and load files. This boundary prevents a common mistake: cleaning the only copy of a file and losing the evidence needed to explain downstream numbers.

The second boundary is the execution environment. Python, Java, native libraries, SQL engines, and warehouse settings can all change behavior. Pinning a lockfile or container image makes dependency resolution repeatable. Recording the command used to run the job makes execution repeatable. The goal is not to freeze the universe forever; it is to make each run attributable to a known environment so upgrades are deliberate events.

The third boundary is deterministic transformation. A deterministic transform produces the same output for the same inputs. Sorting before writing CSV output, fixing time zones, avoiding unseeded randomness, and using stable partition keys are small choices that remove accidental differences. Idempotence is related but distinct: running the same job twice should not duplicate rows, append stale files, or corrupt state. In batch projects, idempotence often means writing to a temporary path and replacing the final artifact only after validation passes.

Project anatomy and configuration

A compact reproducible project usually has a shape like this: src/ for importable code, tests/ for checks, data/raw/ for immutable samples or externally referenced inputs, data/processed/ for rebuildable outputs, artifacts/ for manifests and reports, and a runner such as Makefile, pyproject.toml, or a workflow definition. The important point is not the exact folder names. The important point is that every location has a rule: which files are source, which are generated, which can be deleted, and which must be preserved.

Configuration should be explicit and layered. Defaults live in version control. Environment-specific values, such as a warehouse account or object-store bucket, are supplied by environment variables or deployment configuration. Secrets are never committed. Data-version identifiers, date windows, and run modes should be command arguments or config keys because they change the result. When those values are captured in a manifest, an auditor can distinguish a code change from a parameter change.

Example 1: fingerprint a raw input

The smallest reproducibility checkpoint is a content hash. A path name alone is weak evidence because a file can be replaced without changing its name. A SHA-256 digest identifies the exact bytes used by the run. The example creates a tiny raw orders file and prints its digest.

import hashlib
from pathlib import Path

raw = Path('data/raw/orders.csv')
raw.parent.mkdir(parents=True, exist_ok=True)
raw.write_text('order_id,customer_id,amount\n1,17,42.50\n2,18,13.00\n', encoding='utf-8')

digest = hashlib.sha256(raw.read_bytes()).hexdigest()
print(f'{raw}:{digest}')

Expected behavior: the command prints a line beginning with data/raw/orders.csv: followed by a 64-character hexadecimal digest. If you edit the amount 42.50 to 42.51, the digest changes completely. That sensitivity is useful: a later pipeline can prove whether a downstream output came from the original input or a modified one.

Example 2: build a deterministic transform

The next step is a transform whose output order and formatting are stable. This script groups orders by customer, sorts customers numerically, formats money with two decimal places, and writes a processed CSV. Sorting matters because dictionaries, distributed workers, and database result sets should not be trusted to provide business ordering unless you request it.

from pathlib import Path
import csv

raw_path = Path('data/raw/orders.csv')
out_path = Path('data/processed/customer_totals.csv')
out_path.parent.mkdir(parents=True, exist_ok=True)

totals = {}
with raw_path.open(newline='', encoding='utf-8') as f:
    for row in csv.DictReader(f):
        totals[row['customer_id']] = totals.get(row['customer_id'], 0.0) + float(row['amount'])

with out_path.open('w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['customer_id', 'total_amount'])
    for customer_id in sorted(totals, key=int):
        writer.writerow([customer_id, f'{totals[customer_id]:.2f}'])

print(out_path.read_text(encoding='utf-8'))

Expected output is deterministic for the sample input: customer_id,total_amount, then 17,42.50, then 18,13.00, each on its own line. If the script is run again with the same raw file, it overwrites the same processed file with equivalent content instead of appending duplicate rows. That makes the transform idempotent for this local-file target.

Example 3: record a run manifest

A manifest connects inputs and outputs. In larger systems it may include Git commit, container image, package lock hash, source table snapshots, row counts, schema versions, and validation results. This minimal manifest records the raw and processed file hashes.

from pathlib import Path
import hashlib
import json

paths = [Path('data/raw/orders.csv'), Path('data/processed/customer_totals.csv')]
manifest = {
    str(path): hashlib.sha256(path.read_bytes()).hexdigest()
    for path in paths
}
Path('artifacts').mkdir(exist_ok=True)
Path('artifacts/manifest.json').write_text(json.dumps(manifest, indent=2, sort_keys=True), encoding='utf-8')
print(json.dumps(manifest, indent=2, sort_keys=True))

Expected behavior: artifacts/manifest.json is created with two keys sorted by path. A reviewer can rerun the pipeline, regenerate the manifest, and compare it with the committed or archived manifest. If the processed hash changes while the raw hash and code version did not, the project has a nondeterminism problem or an undocumented environment change.

Design choices and trade-offs

One choice is whether to store data in the repository. Small fixtures belong in version control because they make tests fast and independent. Large raw datasets usually belong in object storage or a data lake, with the repository storing immutable URIs, snapshot IDs, or checksums. Committing large changing files slows reviews and encourages partial clones that hide the actual pipeline state.

Another choice is whether to use scripts, notebooks, or workflow orchestrators. Scripts are easier to run in automation and easier to test. Notebooks are useful for exploration, but production notebook execution needs parameter cells, stripped outputs, and a clear promotion path into tested modules. Orchestrators add retries, schedules, task lineage, and backfill controls, but they do not create reproducibility by themselves. A poorly specified DAG can still read moving inputs and produce unexplained results.

Output format is also a trade-off. CSV is inspectable and portable, but weakly typed. Parquet preserves column types and compresses well, but requires compatible readers. Database tables support access control and concurrent querying, but need migration discipline and snapshot semantics. Pick the format that preserves the guarantees your consumers need, then encode those guarantees in tests.

Failure modes and troubleshooting

Symptom: a rerun produces rows in a different order. Cause: the transform relied on incidental iteration or database order. Diagnose: compare manifests, then diff outputs after sorting by all columns. If sorted files match, ordering is the only nondeterministic part. Correction: add an explicit ORDER BY in SQL or a sort before writing files.

Symptom: totals differ by a few cents across machines. Cause: binary floating-point arithmetic, locale-specific parsing, or dependency changes. Diagnose: print package versions, inspect decimal separators, and add a fixture with known currency totals. Correction: use decimal types for money, pin dependencies, and fail validation when rounded totals drift.

Symptom: running the job twice doubles the loaded records. Cause: an append-only write path was used for a rebuildable batch output. Diagnose: check row counts before and after a second run with unchanged inputs. Correction: write by partition replacement, merge on a stable primary key, or truncate a staging table inside a transaction before loading.

Symptom: a teammate cannot run the project from a clean checkout. Cause: hidden local state such as an uncommitted config file, cached credential, or missing directory. Diagnose: run in a fresh container or temporary directory and record the first missing dependency. Correction: document prerequisites, create directories in code, provide sample configuration, and keep secrets outside the repository.

Security, performance, and reliability implications

Reproducibility improves security because inputs and artifacts become attributable. A checksum does not prove data is trustworthy, but it proves whether the data changed. Manifests can support incident response by showing which downstream outputs were built from a suspect extract. Secrets should be injected at runtime and excluded from manifests; record secret names or connection aliases, not secret values.

Performance work also depends on reproducibility. If the benchmark data, partition range, or dependency set changes between runs, optimization claims are weak. Keep benchmark fixtures stable, record row counts, and separate cold-cache from warm-cache measurements. Reliable recovery uses the same foundation: when outputs are rebuildable from immutable inputs, cleanup can delete partial processed files and rerun rather than attempt manual repair.

Hands-on lab: build and verify a tiny reproducible project

Prerequisites: Python 3, a terminal, and an empty working directory. No external service is required. Create the directories data/raw, data/processed, and artifacts. Save the three example scripts as 01_seed_raw.py, 02_transform.py, and 03_manifest.py.

  1. Run python 01_seed_raw.py. Verify that data/raw/orders.csv exists and that the terminal printed a 64-character digest.
  2. Run python 02_transform.py. Verify that data/processed/customer_totals.csv contains one row per customer and no duplicate header rows.
  3. Run python 03_manifest.py. Verify that artifacts/manifest.json contains hashes for both the raw and processed files.
  4. Run steps 2 and 3 again without editing any input. The processed file and manifest should remain unchanged.
  5. Edit the raw file by adding 3,17,7.50, rerun steps 2 and 3, and verify that customer 17 now has 50.00. The raw and processed hashes should both change.

Cleanup: remove the generated data and artifacts directories if this was a scratch exercise. In a real project, keep raw snapshots or their external immutable references, but regenerated processed outputs can usually be deleted and rebuilt.

Assessment exercises

  1. A pipeline reads latest/orders.csv every morning. Explain why that path is not enough for reproducibility and propose two stronger identifiers.
  2. A Spark job writes partitioned Parquet output but sometimes leaves partial partitions after failure. Design an idempotent write pattern that avoids exposing incomplete results.
  3. You inherit a notebook that depends on a local CSV and a manually installed package. List the changes needed before it can be used as a scheduled data engineering job.
  4. Two manifests have the same raw input hash but different processed output hashes. What hypotheses would you test first, and what evidence would confirm each one?
  5. Choose CSV, Parquet, or a database table for a customer metrics output. Defend the choice using reproducibility, schema, and consumer needs.

Summary

A reproducible data engineering project makes data outputs explainable. It preserves raw inputs or immutable references, pins the runtime, exposes configuration, writes deterministic and idempotent transforms, and records manifests that connect artifacts to exact inputs. These practices turn a pipeline from a one-time local success into a project that can be rebuilt, reviewed, optimized, and trusted during backfills, audits, and incident response.