Medallion Architecture Data Products and Data Mesh

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose and Outcome

Medallion architecture is a lakehouse design pattern that organizes data into bronze, silver, and gold layers. Data mesh is an operating model that gives business domains responsibility for useful, governed data products. This lesson connects the two: the medallion layers describe how data is refined, while the mesh describes who owns the refinement, how consumers discover it, and what guarantees the product makes.

By the end, you should be able to map raw source feeds into bronze tables, clean and conform them into silver tables, publish gold tables as data products, and decide where domain ownership helps or hurts. In a data engineering course, this topic sits between storage design and platform operation: it turns files and tables into reusable analytical interfaces.

How the Layers Work Internally

The bronze layer is the landing zone. It stores source-shaped records with minimal transformation, usually appending ingestion metadata such as source file, load timestamp, batch identifier, and schema version. Bronze is not meant to be beautiful. Its job is replayability: if a downstream rule changes, engineers can rebuild from the original evidence without asking the source system to resend history.

The silver layer applies interpretation. It parses types, standardizes names, removes invalid records from the main path, deduplicates events, resolves simple reference data, and records rejected rows in quarantine tables. Silver tables should have clear grains, such as one row per order event or one current row per customer. The layer is often where late-arriving data, merge keys, and slowly changing dimensions become explicit engineering choices.

The gold layer is consumer-oriented. Gold tables aggregate, join, filter, or reshape silver data for a business purpose: revenue by customer, inventory availability, campaign attribution, or fraud features. A gold table becomes a data product when it has an owner, documented semantics, quality checks, access rules, and a stable interface. Not every gold table deserves product status; temporary analyst work should not be promoted into a shared contract unless it has real consumers.

Data Products and Mesh Ownership

In a mesh, domains own data products close to the business process that creates or understands the data. The commerce domain might own orders and customer revenue; logistics might own shipments; finance might own recognized revenue. A central platform team usually provides storage, orchestration, catalogs, lineage, policy enforcement, and templates, but does not manually curate every domain’s tables.

The internal mechanism is a shift from pipeline ownership by a central team to product ownership by domains using shared platform capabilities. A data product has an interface, such as a table, view, stream, API, or feature set. It also has behavior: refresh cadence, allowed lag, schema compatibility rules, historical backfill policy, and quality expectations. The medallion model gives domains a repeatable path for building those interfaces without inventing a new pipeline shape every time.

Configuration and API Anatomy

A medallion implementation usually defines storage locations, table names, schemas, partitioning, merge keys, and quality checks. The exact syntax differs across Spark, dbt, SQL engines, and lakehouse catalogs, but the anatomy is consistent. Bronze declarations point to sources and ingestion metadata. Silver declarations define parsing and cleansing rules. Gold declarations define business metrics and published grains. Product metadata then declares ownership and consumer guarantees.

Important terms include grain, the uniqueness rule for rows; watermark, the boundary for accepting late events; quarantine, the place for records that failed validation; lineage, the dependency graph from source to product; and SLO, a measurable service objective such as freshness under one hour.

Example 1: Bronze to Silver Orders

This first example keeps raw records intact in bronze, then parses the amount field for silver. One bad amount is sent to quarantine instead of stopping the whole batch. The deterministic output is 2 1 1: two raw rows loaded, one valid silver row, and one rejected row.

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class OrderEvent:
    order_id: str
    customer_id: str
    event_time: str
    amount: float
    source_file: str

raw_events = [
    {"order_id": "A100", "customer_id": "C7", "event_time": "2026-03-01T10:00:00Z", "amount": "19.99", "source_file": "orders-1.json"},
    {"order_id": "A101", "customer_id": "C8", "event_time": "2026-03-01T10:05:00Z", "amount": "bad", "source_file": "orders-1.json"},
]

def bronze_load(rows: Iterable[dict]) -> list[dict]:
    return [{**row, "ingested_at": "2026-03-01T10:10:00Z"} for row in rows]

def silver_orders(rows: Iterable[dict]) -> tuple[list[OrderEvent], list[dict]]:
    valid: list[OrderEvent] = []
    quarantine: list[dict] = []
    for row in rows:
        try:
            valid.append(OrderEvent(
                order_id=str(row["order_id"]),
                customer_id=str(row["customer_id"]),
                event_time=str(row["event_time"]),
                amount=float(row["amount"]),
                source_file=str(row["source_file"]),
            ))
        except (KeyError, TypeError, ValueError) as exc:
            quarantine.append({"row": row, "reason": type(exc).__name__})
    return valid, quarantine

bronze = bronze_load(raw_events)
silver, rejected = silver_orders(bronze)
print(len(bronze), len(silver), len(rejected))

The design point is that bronze accepts source reality, while silver asserts table reality. If the source sends a string where a decimal is expected, the silver step makes that visible. Quarantine also preserves the failed row and reason so the source team can correct the feed or the pipeline can add a documented parsing rule.

Example 2: Silver to Gold Metric

The next example aggregates cleaned orders into customer revenue. This is a gold-shaped table because it answers a consumer question rather than mirroring an operational event. The expected output is [{'customer_id': 'C7', 'revenue': 28.49}, {'customer_id': 'C8', 'revenue': 42.0}].

from collections import defaultdict

silver_orders = [
    {"order_id": "A100", "customer_id": "C7", "amount": 19.99},
    {"order_id": "A102", "customer_id": "C7", "amount": 8.50},
    {"order_id": "A103", "customer_id": "C8", "amount": 42.00},
]

totals = defaultdict(float)
for order in silver_orders:
    totals[order["customer_id"]] += order["amount"]

gold_customer_revenue = [
    {"customer_id": customer_id, "revenue": round(revenue, 2)}
    for customer_id, revenue in sorted(totals.items())
]
print(gold_customer_revenue)

A production version would also include the reporting period, currency, refund handling, and recognition rules. Those details belong in product documentation because two teams can both say revenue while meaning bookings, net revenue, invoiced revenue, or recognized revenue.

Example 3: Gold Table as a Data Product

This product descriptor is a configuration fragment rather than a standalone program. It records who owns the product, where it is served, its grain, its consumers, and the quality objectives consumers can rely on.

data_product:
  name: customer-revenue
  domain: commerce
  owner: commerce-analytics
  serving_layer: gold.customer_revenue
  grain: one row per customer per reporting period
  consumers:
    - finance
    - growth
  quality_slos:
    freshness_minutes: 60
    duplicate_customer_period_rows: 0
  interface:
    fields:
      - name: customer_id
        type: string
      - name: revenue
        type: decimal
      - name: reporting_period
        type: date

The key detail is that the serving layer is only one field. The product is not just gold.customer_revenue; it is the table plus semantics, ownership, checks, and change management. A mesh without these details becomes a collection of domain folders, not a reliable product system.

Example 4: Product Verification

A product should fail before publication when it violates its grain. This verification function detects duplicate customer-period rows. The expected output is ["duplicate grain: ('C7', '2026-03-01')"].

def verify_customer_revenue(rows: list[dict]) -> list[str]:
    errors: list[str] = []
    seen = set()
    for row in rows:
        key = (row.get("customer_id"), row.get("reporting_period"))
        if key in seen:
            errors.append(f"duplicate grain: {key}")
        seen.add(key)
        if row.get("revenue", 0) < 0:
            errors.append(f"negative revenue: {key}")
    return errors

candidate_rows = [
    {"customer_id": "C7", "reporting_period": "2026-03-01", "revenue": 28.49},
    {"customer_id": "C7", "reporting_period": "2026-03-01", "revenue": 30.00},
]
print(verify_customer_revenue(candidate_rows))

Checks like this are more useful when attached to release gates than when left as dashboard warnings. A failed gold build should either keep serving the previous valid table version or publish with an explicit incident, depending on consumer tolerance and platform policy.

Design Choices and Trade-offs

One design choice is whether silver tables are enterprise-wide canonical models or domain-local refined models. Canonical silver reduces duplication and makes joins easier, but it can create a bottleneck because every domain waits for common definitions. Domain-local silver lets teams move quickly, but cross-domain products need stronger documentation and reconciliation rules.

Another choice is how much raw data bronze should retain. Keeping everything improves auditability and backfills, but storage cost, sensitive data exposure, and catalog clutter increase. Many teams use retention tiers: raw bronze for a fixed window, compacted bronze for long-term replay, and stricter masking for regulated fields.

Gold product design also involves trade-offs. A wide product with many metrics is convenient for consumers but harder to evolve without breaking someone. Narrow products are easier to govern and test, but consumers may perform inconsistent joins. The right answer depends on grain, access pattern, and the likelihood that metrics change independently.

Failure Modes and Troubleshooting

Symptom: a dashboard drops yesterday’s revenue to zero. Cause: the gold aggregation ran before late order events arrived or used the wrong watermark. Diagnostics: compare bronze ingestion timestamps with event timestamps, inspect the orchestrator run order, and count silver rows by event date and load date. Correction: extend the late-arrival window, reprocess the affected partitions, and publish the recovered gold partition after checks pass.

Symptom: downstream joins suddenly duplicate rows. Cause: a silver dimension changed grain from one current row per customer to multiple rows per customer without consumers adapting. Diagnostics: run uniqueness checks on the documented key, inspect recent schema or transformation changes, and trace lineage to the first table where duplicates appear. Correction: restore the original grain, publish a separate history table for multiple versions, and update product metadata before consumers switch.

Symptom: a domain product passes freshness checks but contains stale business logic. Cause: the table refreshed on time, but a reference mapping was not updated. Diagnostics: compare reference data effective dates, sample records around the business rule change, and review the product owner’s release notes. Correction: backfill silver enrichment, rerun gold products that depend on it, and add a rule-specific quality check rather than relying only on freshness.

Security, Performance, and Reliability

Security should follow the data as it moves through layers. Bronze often contains the most sensitive source-shaped payloads, so access is usually limited to pipeline operators and domain maintainers. Silver may tokenize or mask fields. Gold products should expose only the columns and rows required by their consumers, with policies enforced through the catalog or serving engine rather than by convention.

Performance improves when each layer has a physical design that matches its access pattern. Bronze benefits from append-friendly layout and ingestion-date partitioning. Silver often needs merge performance on business keys and event dates. Gold products should be clustered or materialized for common filters, such as reporting period or domain identifier. Excessive layering can add latency, so teams should measure whether a table adds useful semantics or just another dependency.

Reliability depends on replayable inputs, idempotent transformations, and versioned publication. Rebuilding silver from bronze should produce the same rows for the same rules. Publishing gold should be atomic from the consumer perspective: consumers should see either the previous valid product or the new valid product, not a half-written aggregate.

Hands-on Lab: Build a Mini Product

Prerequisites: a local Python interpreter and a text editor. No external services are required. Create a temporary working directory for the lab.

  1. Create a small list of raw order dictionaries with one valid row, one duplicate order identifier, and one invalid amount.
  2. Write a bronze function that appends source_file and ingested_at metadata without changing source fields.
  3. Write a silver function that casts amounts to floats, rejects invalid rows, and deduplicates by order_id using the latest event time.
  4. Write a gold function that groups silver rows by customer_id and reporting date.
  5. Add a product verification step that checks customer-date uniqueness, non-negative revenue, and row count greater than zero.

Verification: run the script twice with the same input. The bronze row count should equal the raw row count each run, the silver count should exclude invalid duplicates according to your chosen rule, and the gold verifier should return an empty error list. Then change one gold row to a negative amount and confirm the verifier reports an error.

Cleanup: delete the temporary directory. If you used a real lakehouse environment instead of local Python, drop only the lab tables or database you created and confirm no shared catalog objects point to them.

Assessment Exercises

  1. A source sends order updates and deletes. Which fields must bronze preserve so silver can rebuild the correct current order state?
  2. Your finance team and growth team both consume customer revenue but disagree about refunds. Should you publish one gold product or two? Explain using grain, semantics, and ownership.
  3. A silver customer table starts failing uniqueness checks after a marketing system is added. Describe the most likely grain problem and how you would diagnose it.
  4. Design a freshness SLO for a shipment-status product. What layer would you measure, and why?
  5. Identify one case where adding another medallion table would make the platform less reliable rather than more reliable.

Summary

Medallion architecture gives data engineers a concrete refinement path: bronze captures source evidence, silver applies reliable interpretation, and gold serves consumer use cases. Data products add ownership, documented semantics, quality objectives, and access rules. Data mesh scales the model by making domains accountable for products while the platform supplies shared tools. The strongest designs keep layer purpose, table grain, and product guarantees explicit, then verify those guarantees before consumers depend on them.