Data Contracts Schemas and Evolution

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: a shared contract for changing data

A data contract is an explicit agreement for a data product: the fields a producer will publish, the meanings of those fields, the allowed values, freshness expectations, ownership, and the rules for changing the shape over time. In data engineering, the outcome is not merely a nicer schema document. The outcome is that a downstream model, dashboard, feature store, or reverse ETL job can decide whether a new dataset version is safe before it consumes it.

This lesson focuses on schemas and evolution. A schema describes structure: column names, data types, nullability, constraints, nested fields, and sometimes semantic annotations. Evolution describes how that structure changes without surprising consumers. A contract turns both into a controlled interface. Instead of discovering after a warehouse load that customer_id silently changed from an integer to a string, the contract check fails at the producer boundary, in CI, or during ingestion with a reason that points to the incompatible change.

How contract enforcement works internally

A practical contract system has four moving parts. First, a canonical contract document stores the intended interface. It may be YAML, JSON Schema, Avro schema, Protobuf IDL, a table schema in a catalog, or a purpose-built manifest. Second, validators compare actual records or table metadata with that document. Third, a compatibility checker compares a proposed new contract with the currently published one. Fourth, registries or catalogs make the accepted contract discoverable by consumers and automation.

Validation usually happens at two layers. Record-level validation checks values: required fields are present, types match, enum values are allowed, and constraints such as minimum length or nonnegative amounts hold. Metadata-level validation checks table definitions: column names, physical types, partition fields, primary keys, and ownership. Streaming systems often validate serialized messages before they land on a topic. Batch systems often validate files or DataFrame schemas before writing to object storage or a warehouse.

Evolution is governed by compatibility direction. Backward compatibility means a new consumer using the new schema can read data written with the old schema. Forward compatibility means an old consumer can read data written with the new schema. Full compatibility means both directions hold. The exact rules differ by format. In many schema systems, adding an optional field with a default is compatible, removing a required field is breaking, narrowing a type is risky, and renaming is treated as remove plus add unless aliases are supported.

Contract anatomy

A useful data contract separates technical shape from data semantics. Technical fields include name, type, required or nullable status, default value, and constraints. Semantic fields include description, business meaning, unit, data classification, owner, and deprecation status. Operational fields define service-level expectations: freshness, delivery cadence, uniqueness keys, partitioning, and quality checks.

from dataclasses import dataclass

@dataclass(frozen=True)
class Field:
    name: str
    dtype: str
    required: bool = True

CUSTOMER_V1 = [
    Field("customer_id", "string"),
    Field("email", "string"),
    Field("marketing_opt_in", "boolean", required=False),
]

print([field.name for field in CUSTOMER_V1])

This small Python representation is not a replacement for a registry, but it shows the minimum anatomy: each field has a stable name, a declared type, and a required flag. The expected output is ['customer_id', 'email', 'marketing_opt_in']. In a real platform the same ideas might live in an Avro schema, dbt model contract, Great Expectations suite, or OpenAPI-like data product manifest.

Example 1: validating a producer payload

Start with a producer that publishes customer records. The contract requires customer_id and email, allows optional marketing_opt_in, and rejects unknown fields. Rejecting unknown fields is strict, but it prevents producers from accidentally launching fields consumers may treat as official.

CONTRACT = {
    "customer_id": {"type": str, "required": True},
    "email": {"type": str, "required": True},
    "marketing_opt_in": {"type": bool, "required": False},
}

record = {"customer_id": "c-100", "email": "ada@example.com"}

errors = []
for name, rule in CONTRACT.items():
    if rule["required"] and name not in record:
        errors.append(f"missing {name}")
    if name in record and not isinstance(record[name], rule["type"]):
        errors.append(f"wrong type for {name}")
for name in record:
    if name not in CONTRACT:
        errors.append(f"unknown field {name}")

print("valid" if not errors else errors)

The deterministic output is valid. If email is removed, the validator returns ['missing email']. If the producer sends marketing_opt_in as 'yes', it returns ['wrong type for marketing_opt_in']. The mechanism is intentionally boring: contract checks are just comparisons between observed data and declared rules. Their value comes from placing that comparison before data spreads.

Example 2: checking whether a schema change is compatible

Now evolve the customer contract. A common compatible change is adding an optional field such as country_code. A common breaking change is making a previously optional field required, because old records do not have it. The following simplified checker encodes those rules.

old = {
    "customer_id": {"type": "string", "required": True},
    "email": {"type": "string", "required": True},
}

new = {
    "customer_id": {"type": "string", "required": True},
    "email": {"type": "string", "required": True},
    "country_code": {"type": "string", "required": False},
}

def compatibility_errors(previous, proposed):
    errors = []
    for name, rule in previous.items():
        if name not in proposed:
            errors.append(f"removed field: {name}")
        elif proposed[name]["type"] != rule["type"]:
            errors.append(f"changed type: {name}")
        elif not rule["required"] and proposed[name]["required"]:
            errors.append(f"made optional field required: {name}")
    for name, rule in proposed.items():
        if name not in previous and rule["required"]:
            errors.append(f"added required field: {name}")
    return errors

print(compatibility_errors(old, new))

The expected output is [], meaning the proposed schema passes this compatibility policy. If country_code is required, the checker reports ['added required field: country_code']. If email is removed, it reports ['removed field: email']. Production tools add more nuance, such as numeric widening, aliases, logical timestamp types, default values, and nested-field rules, but the internal question is the same: can old and new readers and writers coexist during rollout?

Example 3: warehouse contract with semantic quality checks

Contracts should not stop at types. A warehouse table can have the right columns and still be useless if keys duplicate or timestamps arrive too late. In this example, the contract requires unique order IDs, nonnegative totals, and a fresh extract date. These checks are often implemented in SQL tests, orchestration assertions, or table-quality frameworks.

from datetime import date

orders = [
    {"order_id": "o-1", "total_usd": 42.50, "extract_date": date(2026, 9, 7)},
    {"order_id": "o-2", "total_usd": 0.0, "extract_date": date(2026, 9, 7)},
]

def check_orders(rows, expected_date):
    ids = [row["order_id"] for row in rows]
    errors = []
    if len(ids) != len(set(ids)):
        errors.append("order_id is not unique")
    if any(row["total_usd"] < 0 for row in rows):
        errors.append("total_usd contains negative values")
    if any(row["extract_date"] != expected_date for row in rows):
        errors.append("extract_date is stale")
    return errors

print(check_orders(orders, date(2026, 9, 7)))

The output is []. If a duplicate order_id appears, downstream revenue aggregation may double count. If the date is stale, a daily executive dashboard can show yesterday’s results as though they are current. Schema, quality, and freshness belong together because consumers experience them as one interface.

Design choices and trade-offs

Strict contracts catch problems early but require coordination. A strict policy that rejects unknown fields is useful for regulated or heavily shared datasets, while a permissive policy that ignores unknown optional fields can speed experimentation. The choice should follow consumer risk, not producer convenience.

Schema format also matters. Avro and Protobuf are strong for event streams because they support compact binary encoding and schema registry compatibility checks. JSON Schema is approachable for APIs and semi-structured payloads. Warehouse contracts are often expressed as SQL DDL plus tests because the physical table is the interface. Using one universal format across all layers sounds attractive, but teams often get better results by mapping a common contract model into layer-specific validators.

Versioning is another trade-off. A major version can mark breaking changes, such as replacing customer_id with party_id. Minor versions can add optional fields or relax constraints. Deprecation windows reduce disruption: publish the new field, populate both old and new fields, notify owners, measure consumer migration, then remove the old field only after the agreed date.

Failure modes and troubleshooting

Symptom: a streaming job starts sending records to a dead-letter topic after a deployment. Cause: the producer changed event_time from an ISO string to an epoch integer while the registered contract still expects a string. Diagnostic steps: sample a failed message, compare it with the active contract version in the registry, and inspect the deploy diff for serialization changes. Correction: roll back the serializer or publish a planned compatible field such as event_time_epoch_ms while keeping event_time stable.

Symptom: a dbt or warehouse build fails on a contract test after a source migration. Cause: the source column total_usd now arrives nullable, but the contract declares it required. Diagnostic steps: count nulls by load batch, inspect the upstream migration notes, and confirm whether null means missing, zero, or not applicable. Correction: fix the source mapping if nulls are accidental, or deliberately evolve the contract and downstream logic if null is a legitimate business state.

Symptom: consumers complain that a supposedly compatible field addition broke their parser. Cause: the consumer uses a rigid deserializer that rejects unknown fields even though the contract policy permits adding optional fields. Diagnostic steps: reproduce with the consumer’s parser, check whether it is configured for strict unknown-field handling, and compare against the published compatibility policy. Correction: update the consumer parser, or tighten the producer policy so future additions require consumer readiness.

Reliability, performance, and security implications

Contracts improve reliability by making interface breakage observable at controlled points. They also add cost. Validating every record in a high-volume stream can increase latency, so teams often validate at serialization time, sample records, or separate cheap type checks from heavier quality checks. Batch pipelines can run complete validation before publishing a partition, which avoids exposing partial bad data.

Security labels belong in contracts when fields carry personal, payment, health, or confidential business data. A field such as email can be marked as personal data, which allows automated policy checks to block replication to unauthorized zones or require hashing before analytics use. The contract is not the access-control system, but it supplies the metadata that access-control and lineage systems need.

Hands-on lab: add a compatible field safely

Prerequisites: Python 3, a terminal, and the ability to run a single local script. No external service is required. The lab simulates a contract review for a customer event.

  1. Create a file named contract_lab.py with the compatibility checker from Example 2.
  2. Run it once with country_code marked optional and confirm the output is [].
  3. Change country_code to required and run it again.
  4. Add a default value to your proposed field in a separate metadata key, then decide whether your real platform treats that as compatible. Document the policy in the contract rather than relying on memory.
  5. Rollback by returning country_code to optional, then rerun the script.

Verification succeeds when the optional addition produces no errors and the required addition reports ['added required field: country_code']. Cleanup is simply deleting the local lab file. In a real repository, cleanup may mean closing the schema-change branch or reverting an unpublished contract proposal.

Assessment exercises

  1. A producer wants to rename user_id to account_id. Explain why many contract systems treat this as a breaking change and design a two-release migration plan.
  2. You discover that a field declared as non-null is null in 3 percent of records. What evidence would you gather before deciding whether to fix the producer or evolve the contract?
  3. For a high-volume clickstream, where would you place validation: producer, registry, ingestion job, warehouse load, or all of them? Justify the latency and reliability trade-offs.
  4. Write compatibility rules for adding, removing, and changing nested fields in an event payload consumed by mobile apps that update slowly.
  5. Choose whether unknown fields should be allowed for a regulated finance dataset. Explain the consumer and governance consequences.

Summary

Data contracts make schemas and evolution an explicit engineering interface. The core mechanism is comparison: records and table metadata are checked against a declared contract, while proposed contract versions are checked against compatibility rules before release. Strong contracts combine field shape, semantic meaning, quality expectations, ownership, and a disciplined migration process. Used well, they let data products change without making downstream systems guess what changed or why.