Dataset Versioning and Lineage

Dataset versioning and lineage answer two operational questions that ordinary file names cannot answer: exactly which data trained or evaluated this model, and how was that data produced? In this lesson, the outcome is a repeatable way to identify datasets, record their parents and transformations, and trace a deployed model back to the raw sources, feature tables, code revision, and validation checks that shaped it.

Within an MLOps pipeline, model behavior depends on mutable inputs. Labels arrive late, source systems backfill records, feature jobs change, and engineers rerun experiments. Dataset versioning freezes a named view of data so training and evaluation can be reproduced. Lineage records the directed chain from sources to derived assets so a team can audit, debug, compare, and rollback with evidence instead of guesswork.

How the Mechanism Works

A dataset version is usually an immutable reference, not a copy pasted into a new folder by hand. The reference may point to object storage paths plus checksums, a table snapshot, a lakehouse commit, a DVC-style content hash, or a feature-store materialization timestamp. The important property is identity: reading the same version later must produce the same logical rows, columns, and values, or the system must detect that the data changed.

Lineage is a graph. Nodes represent assets such as raw files, source tables, cleaned datasets, feature sets, labels, validation reports, model artifacts, and deployed endpoints. Edges represent dependencies: this feature table was built from those source partitions using this transform; this model was trained from that feature version with these hyperparameters and code revision. The graph is normally a directed acyclic graph because derived assets should point backward to earlier inputs rather than forming loops.

The storage layer and metadata layer play different roles. The storage layer keeps bytes or table snapshots addressable. The metadata layer stores the manifest: dataset name, version identifier, schema, row count, partition range, source references, transform command, code revision, author or service identity, creation time, quality checks, and retention policy. A model registry can then link a model version to the exact dataset manifest used for training and to a separate manifest used for evaluation.

Manifest Anatomy

Field Purpose
dataset Stable logical name, such as orders_training, independent of one file path.
version Immutable identifier such as a content hash, table commit, or snapshot id.
parents Input assets used to create this dataset version.
transform Command, pipeline step, notebook export, or job id that produced the asset.
schema Column names, types, and sometimes semantic constraints.
checks Completeness, freshness, uniqueness, leakage, and distribution results.

Good manifests separate logical names from physical locations. A physical path may change when storage is compacted or replicated; a version should still identify the same dataset. Conversely, a mutable table name such as customer_features_latest is not enough for training evidence because its contents may differ tomorrow.

Example 1: Content Addressing a Raw File

The smallest useful version is a digest of the bytes being consumed. This example hashes a tiny CSV payload and prints a SHA-256 based identifier. The exact digest is deterministic for the payload contents, while the byte count confirms which data was hashed.

import hashlib

payload = (
    b"order_id,customer_id,amount,created_at\n"
    b"1,C001,12.50,2026-01-04\n"
    b"2,C002,8.00,2026-01-04\n"
)

version_id = "sha256:" + hashlib.sha256(payload).hexdigest()
print(version_id)
print(len(payload))

Expected output is sha256:96e96a11b07507501e0aa1ba8339a042e340c7db70f1de4609a13b6c57e594c9 and then 86. If any value, delimiter, newline, or encoding changes, the digest changes. That makes accidental overwrites visible, but it does not by itself describe the schema, source owner, or business meaning of the rows.

Example 2: Recording a Dataset Manifest

A digest becomes more useful when wrapped in a manifest. The manifest below records the logical dataset name, the raw parent asset, the transform command, the schema, and the row count. The validation check fails fast if required metadata is missing.

import json

manifest = {
    "dataset": "orders_training",
    "version": "sha256:96e96a11b07507501e0aa1ba8339a042e340c7db70f1de4609a13b6c57e594c9",
    "parents": ["raw/orders.csv@sha256:96e96a11b07507501e0aa1ba8339a042e340c7db70f1de4609a13b6c57e594c9"],
    "transform": "python make_features.py --window-days 30",
    "schema": {"order_id": "int", "amount": "float", "created_at": "date"},
    "row_count": 2,
}

required = {"dataset", "version", "parents", "transform", "schema", "row_count"}
missing = required.difference(manifest)
if missing:
    raise ValueError(f"manifest is missing {sorted(missing)}")
print(json.dumps(manifest, sort_keys=True))

Expected behavior: the script prints a JSON object with keys sorted alphabetically. In a real pipeline, this manifest would be stored in a metadata repository and linked to the training run. The manifest is also where you attach validation results: for example, order_id uniqueness, nonnegative amount, allowed event dates, and a check that labels were not joined from the future.

Example 3: Querying Model Lineage

Lineage lets an operator start from a model and walk backward to everything that influenced it. This example represents a small graph as a dictionary and prints all ancestors of a model artifact.

lineage = {
    "raw/orders.csv@sha256:96e96a": [],
    "features/orders_30d.parquet@sha256:8aa1": ["raw/orders.csv@sha256:96e96a"],
    "models/churn.pkl@sha256:4fe2": [
        "features/orders_30d.parquet@sha256:8aa1",
        "code/train.py@git:91ab2c",
    ],
}

def ancestors(node: str) -> set[str]:
    seen = set()
    stack = list(lineage.get(node, []))
    while stack:
        parent = stack.pop()
        if parent in seen:
            continue
        seen.add(parent)
        stack.extend(lineage.get(parent, []))
    return seen

for item in sorted(ancestors("models/churn.pkl@sha256:4fe2")):
    print(item)

Expected output is three lines: code/train.py@git:91ab2c, features/orders_30d.parquet@sha256:8aa1, and raw/orders.csv@sha256:96e96a. The operator can now answer whether a bad raw file reached the deployed model, whether retraining used the intended code revision, and which derived assets need rebuilding after a source correction.

Design Choices and Trade-offs

Content hashes are simple and strong for files, but large datasets make full hashing expensive unless manifests store per-object or per-partition hashes. Table snapshots are efficient for warehouses and lakehouse formats because they reuse storage and expose snapshot ids, but reproducibility depends on retention settings and access to historical metadata. Timestamp-based versions are easy to understand, but they are fragile when late-arriving data or timezone boundaries change the selected records.

Full copies maximize isolation and make rollback straightforward, but they increase storage cost. Delta storage and table snapshots reduce cost, but compaction, vacuum, or lifecycle policies can delete old files if retention is not aligned with audit requirements. Partition-level versioning improves incremental pipelines, yet it adds complexity when a training set spans many partitions with different correction histories.

Lineage can be coarse or fine grained. Coarse lineage, such as model A used feature table B, is cheap and often enough for release review. Column-level lineage is better for privacy analysis and feature debugging, but it requires parsers or instrumented pipeline frameworks and can be incomplete when arbitrary Python code transforms data. Choose the granularity that supports concrete decisions: rebuild scope, incident impact, compliance evidence, and experiment comparison.

Failure Modes and Troubleshooting

Symptom: a model cannot be reproduced even though the training script and random seed are known. Cause: the pipeline used latest source tables, so reruns read corrected or newly arrived rows. Diagnose: compare the training run metadata with warehouse query history and source table commit times. Correct: train from explicit snapshot ids or partition manifests and reject runs that do not include data versions.

Symptom: offline metrics improve sharply, but online performance drops. Cause: the labeled dataset lineage includes features computed after the prediction timestamp, creating leakage. Diagnose: inspect lineage edges and feature definitions for joins that use label time, fulfillment time, or future aggregates. Correct: make point-in-time joins part of the feature generation contract and store the as-of timestamp in the manifest.

Symptom: an incident review cannot determine which customers were affected by a corrupted source batch. Cause: lineage stops at the cleaned table and does not preserve raw partition identifiers. Diagnose: query manifests for parent assets and look for missing partition or batch ids. Correct: record parent versions at the partition or batch level for sources that can be partially repaired.

Symptom: an old model version points to a dataset that no longer exists. Cause: object lifecycle cleanup removed files used by historical snapshots. Diagnose: resolve the manifest paths and check storage retention logs. Correct: align retention with model rollback and audit windows, or archive promoted training and evaluation datasets separately.

Security, Performance, and Reliability

Dataset lineage is sensitive metadata. It may reveal source systems, customer segments, regulated attributes, or incident scope. Store manifests with the same access discipline used for model and data platforms: least-privilege writers, reviewed readers, tamper-evident history, and redaction for fields that should not be exposed in general experiment dashboards.

Performance problems usually come from treating versioning as a full-copy operation for every run. Prefer snapshotting, partition manifests, and content-addressed reuse where possible. Reliability depends on atomic publication: write data first, validate it, then publish the manifest as the single discoverable version. Consumers should read only published manifests, never half-written output folders.

Hands-on Lab: Version a Tiny Training Dataset

Prerequisites: Python 3, a clean working directory, and permission to create scratch files if you choose to persist the examples. The lab uses local data so the mechanics are visible without a cloud account.

  1. Run the first example to compute a content version for the sample CSV payload.
  2. Create a manifest like the second example, replacing the example digest if you use your own file contents.
  3. Add a derived asset name such as features/orders_30d.parquet and record orders.csv@your_digest as its parent.
  4. Add a model node that depends on the feature asset and a fake code revision such as git:local-lab.
  5. Run the lineage query and verify that the raw data, feature asset, and code revision all appear as ancestors of the model.
  6. Change one amount in the payload, recompute the digest, and verify that the new digest differs from the original.

Verification: the manifest contains a dataset name, immutable version, parent list, transform description, schema, and row count; the lineage query reaches every upstream asset; and changed data produces a different version id. Cleanup: delete any scratch CSV or manifest files created during the lab, or keep them in a directory excluded from production pipelines.

Assessment Exercises

  1. A training job records a Git commit and model artifact hash, but no dataset version. Explain which incident questions remain unanswerable and what metadata you would require before promotion.
  2. You have daily source partitions and a weekly training set. Design a manifest structure that supports rebuilding only the weeks affected by one corrected daily partition.
  3. A team proposes using features_latest for evaluation because it is convenient. Describe two ways this can invalidate model comparison and how to enforce a safer reference.
  4. Given coarse table-level lineage and a privacy deletion request for one source column, decide whether the lineage is sufficient. State what extra granularity might be needed and why.
  5. Choose between full dataset copies and snapshot-based versioning for a high-volume feature table. Name the reliability benefit and the operational cost of your choice.

Summary

Dataset versioning gives data a stable identity; lineage explains where that identity came from and what depended on it. In MLOps, those two capabilities make experiments comparable, promotions auditable, leakage diagnosable, and rollback practical. The strongest implementations publish immutable manifests, link them to runs and model versions, retain the underlying snapshots, and make missing lineage a release-blocking error.