Data Quality Dimensions Tests and Observability

Published by Programming Line in Data Engineering in Depth. Last updated 7th September 2026.

Purpose: quality signals that engineers can act on

Data quality dimensions give data engineers a shared vocabulary for deciding whether a dataset is fit for a specific downstream use. A dashboard, machine learning feature table, regulatory extract, and reverse ETL sync can all be technically valid files while still failing users in different ways. Dimension-based testing separates those failure types: completeness asks whether required data is present, validity asks whether values obey rules, uniqueness asks whether entities are duplicated, consistency asks whether related facts agree, timeliness asks whether data arrived within its service window, and accuracy asks whether values match a trusted reference closely enough for the business decision.

The outcome of this lesson is a working mental model for turning quality dimensions into tests and observability signals. In a data engineering system, tests are explicit assertions executed at a known point in the pipeline. Observability is the continuous measurement of data behavior over time, usually with metrics, logs, lineage, incidents, and ownership attached. The two work together: tests stop or quarantine known bad data; observability helps find drift, silent degradation, and failures that no one knew to assert yet.

How dimension-based quality works internally

A quality check has four parts: a subject, a measurement, a rule, and an action. The subject is the dataset, column, partition, stream window, or business entity being examined. The measurement is a computed value such as null count, duplicate count, maximum event age, accepted-value ratio, or row count difference against a source. The rule compares that measurement with an expectation. The action records the result, alerts a team, blocks publication, writes rejects, or annotates downstream lineage.

Internally, most data quality frameworks compile checks into queries or in-memory scans. A completeness check for customer_id becomes a count of rows where customer_id is null. A uniqueness check becomes a grouping query that counts keys with more than one row. A validity check becomes a predicate such as amount >= 0 or status in (...). Timeliness often uses metadata and event-time comparisons rather than only row values: a job can compare the latest partition, maximum event timestamp, ingestion watermark, and scheduler deadline.

The important design detail is granularity. A table-level check can tell you that duplicate order IDs exist, but a partition-level check can tell you duplicates started in the 2026-09-06 daily load. A column-level check can show that email validity dropped from 99.2 percent to 83 percent. An entity-level check can show which merchants, devices, or regions are affected. The more precise the granularity, the faster the diagnosis, but the more checks you must compute and store.

Observability systems persist the measurements, not just pass or fail states. A null-rate metric over thirty days lets you distinguish a new break from normal seasonality. A row-count metric tied to lineage lets you see whether a source outage propagated into curated tables. A freshness metric tied to scheduler runs lets you identify whether late data comes from extraction, transformation, or publication. Good observability records include dataset name, check name, dimension, severity, owner, execution time, measured value, threshold, sample failed rows where safe, and a link to the producing job.

Check anatomy and configuration

A practical quality rule should be readable by data producers and precise enough for automation. Name the dimension, the subject, the expectation, the evaluation point, and the failure action. For example: completeness on orders.customer_id, evaluated after staging load, requires null rate equal to zero, and blocks promotion to the curated schema. A less severe observability monitor might allow a warning when daily row count changes by more than 35 percent compared with the trailing median.

from collections import Counter

orders = [
    {"order_id": "o1", "customer_id": "c1", "amount": 24.50, "status": "paid"},
    {"order_id": "o2", "customer_id": None, "amount": 13.00, "status": "paid"},
    {"order_id": "o2", "customer_id": "c3", "amount": -4.00, "status": "refunded"},
]

required_columns = ["order_id", "customer_id", "amount", "status"]
allowed_statuses = {"paid", "pending", "cancelled", "refunded"}

null_customer_ids = sum(1 for row in orders if row["customer_id"] is None)
invalid_amounts = [row["order_id"] for row in orders if row["amount"] < 0]
invalid_statuses = [row["order_id"] for row in orders if row["status"] not in allowed_statuses]
order_id_counts = Counter(row["order_id"] for row in orders)
duplicate_order_ids = sorted(order_id for order_id, count in order_id_counts.items() if count > 1)

print({
    "schema_columns_present": all(col in orders[0] for col in required_columns),
    "null_customer_ids": null_customer_ids,
    "invalid_amounts": invalid_amounts,
    "invalid_statuses": invalid_statuses,
    "duplicate_order_ids": duplicate_order_ids,
})

This first example checks four dimensions on a tiny order batch. Completeness fails because one row has no customer_id. Validity fails because a refunded row has a negative amount; a real model might allow negative refund transactions only if the schema has a separate transaction type. Uniqueness fails because o2 appears twice. Schema presence passes because the expected columns exist. The deterministic output is a dictionary with one null customer, one invalid amount for o2, no invalid statuses, and duplicate order ID o2.

Progressive example: thresholds and severity

Not every failed measurement should stop a pipeline. A missing primary key in a fact table usually deserves a hard failure. A mild row-count anomaly may deserve a warning until an owner confirms whether a campaign, holiday, or upstream change explains it. Severity should represent consumer impact, not engineering convenience. The same dimension can have different thresholds at different layers: raw ingestion may tolerate missing optional fields, while curated analytics tables usually enforce stricter rules.

def evaluate_threshold(name, measured, operator, threshold, severity):
    passed = {
        "==": measured == threshold,
        "<=": measured <= threshold,
        ">=": measured >= threshold,
    }[operator]
    return {"check": name, "passed": passed, "measured": measured, "threshold": threshold, "severity": severity}

checks = [
    evaluate_threshold("customer_id_null_count", 1, "==", 0, "critical"),
    evaluate_threshold("daily_row_count_minimum", 9820, ">=", 9000, "warning"),
    evaluate_threshold("duplicate_order_id_count", 1, "==", 0, "critical"),
]

for check in checks:
    action = "block" if not check["passed"] and check["severity"] == "critical" else "record"
    print(check["check"], check["passed"], action)

This example separates the measurement from the action. The null-count and duplicate-count checks are critical and would block publication when they fail. The row-count minimum passes and is recorded as evidence. In a production orchestrator, the same result would usually be written to a quality results table and attached to the run ID, dataset version, and lineage graph. Expected output lines are customer_id_null_count False block, daily_row_count_minimum True record, and duplicate_order_id_count False block.

Progressive example: freshness observability

Freshness is a timeliness dimension. It asks whether data is recent enough for its consumers, not whether the job merely succeeded. A pipeline can finish successfully while publishing yesterday’s partition twice, ingesting no new events, or loading records whose event timestamps lag behind the extraction time. Freshness checks commonly compare the maximum event time with the expected deadline and compare observed partitions with a calendar or schedule.

from datetime import datetime, timezone

watermark = datetime(2026, 9, 7, 8, 45, tzinfo=timezone.utc)
deadline = datetime(2026, 9, 7, 9, 0, tzinfo=timezone.utc)
max_event_time = datetime(2026, 9, 7, 8, 40, tzinfo=timezone.utc)
expected_partition = "2026-09-07"
loaded_partition = "2026-09-07"

freshness_minutes = int((watermark - max_event_time).total_seconds() / 60)
passed = watermark <= deadline and loaded_partition == expected_partition and freshness_minutes <= 30

print({"freshness_minutes": freshness_minutes, "passed": passed})

The output is {'freshness_minutes': 5, 'passed': True}. This check would fail if the loaded partition were still 2026-09-06, if the job completed after the 09:00 UTC deadline, or if the newest event were more than thirty minutes behind the ingestion watermark. Notice that this is not a generic job-status check. It inspects data time, scheduler time, and partition identity together.

Design choices and trade-offs

The first trade-off is blocking versus monitoring. Blocking checks protect consumers from known severe defects, but overuse can create fragile pipelines that stop for harmless variance. Monitoring checks preserve availability and trend data, but they require alert routing and human response. Mature systems use both: critical entity keys, referential integrity, and irreversible external exports are blocked; distribution changes, optional-field drift, and row-count anomalies are often monitored first.

The second trade-off is absolute thresholds versus adaptive thresholds. Absolute thresholds are understandable: zero nulls, no duplicate order IDs, status values must be from an allowed set. Adaptive thresholds compare against recent history, seasonality, or peer datasets. They are better at catching unusual volume and distribution shifts, but they can normalize slow degradation if no one reviews the baseline.

The third trade-off is where checks run. Source-adjacent checks catch issues early and give producers fast feedback. Warehouse checks see integrated data and can validate joins, dimensions, and business rules. Consumer-adjacent checks catch final semantic breakage, such as a dashboard aggregate that no longer matches finance totals. Running checks at every layer improves diagnosis but increases compute cost and operational noise.

Failure modes and troubleshooting

A common symptom is a sudden spike in null rates for a required column. The likely causes are source schema changes, extraction bugs, or transformations that changed join behavior. Diagnose by comparing raw, staged, and curated null rates for the same run ID; inspect schema-change logs; sample failed rows; and check whether the nulls are concentrated in one source partition or producer. Correct by updating the extractor mapping, fixing the join condition, backfilling affected partitions, and adding a schema or completeness check earlier in the pipeline.

Another symptom is a freshness alert even though the orchestrator shows green. Causes include a repeated old partition, stalled upstream event production, timezone conversion errors, or checking processing time instead of event time. Diagnose by printing the latest partition name, maximum event timestamp, ingestion watermark, and scheduler completion timestamp side by side. Correct by using event-time watermarks, validating expected partition names, and making the job fail when it publishes an old partition as current.

A third symptom is noisy row-count alerts every Monday. The cause may be legitimate weekly seasonality rather than a defect. Diagnose by plotting row counts by weekday and comparing against the same weekday trailing median. Correct by using weekday-aware thresholds or a calendar-aware model, and keep an absolute lower bound for source outages.

Security, performance, and reliability implications

Quality observability can expose sensitive data if failed-row samples include personal information. Store only the columns needed for diagnosis, mask direct identifiers, and apply the same access controls as the underlying dataset. For performance, prefer aggregated metrics over full failed-row materialization unless debugging requires samples. On large fact tables, partition pruning, approximate distinct counts, and incremental checks often reduce cost while preserving useful signals. Reliability improves when quality results are durable: a failed check should leave an audit record even if the orchestrator retries or rolls back the transformed table.

Hands-on lab: build a local quality report

Prerequisites: a Python interpreter and permission to run a local script. The lab uses only the standard library. Step 1: create a small list of order dictionaries with at least one null customer, one duplicate order ID, and one stale event timestamp. Step 2: compute completeness, uniqueness, validity, and freshness measurements. Step 3: evaluate each measurement against a threshold. Step 4: print a report that includes dimension, measured value, pass or fail, and action. Step 5: correct the bad rows and rerun the script.

from collections import Counter
from datetime import datetime, timezone

orders = [
    {"order_id": "a1", "customer_id": "c1", "amount": 10, "event_time": "2026-09-07T08:50:00+00:00"},
    {"order_id": "a2", "customer_id": None, "amount": 20, "event_time": "2026-09-07T08:51:00+00:00"},
    {"order_id": "a2", "customer_id": "c2", "amount": -5, "event_time": "2026-09-07T07:10:00+00:00"},
]

now = datetime(2026, 9, 7, 9, 0, tzinfo=timezone.utc)
order_counts = Counter(row["order_id"] for row in orders)
max_event_time = max(datetime.fromisoformat(row["event_time"]) for row in orders)
freshness_minutes = int((now - max_event_time).total_seconds() / 60)

report = [
    ("completeness", "null customer_id count", sum(row["customer_id"] is None for row in orders), 0, "block"),
    ("uniqueness", "duplicate order_id count", sum(count > 1 for count in order_counts.values()), 0, "block"),
    ("validity", "negative amount count", sum(row["amount"] < 0 for row in orders), 0, "block"),
    ("timeliness", "freshness minutes", freshness_minutes, 30, "alert"),
]

for dimension, label, measured, threshold, action in report:
    passed = measured <= threshold
    print(f"{dimension}: {label}={measured}, passed={passed}, action={action if not passed else 'record'}")

Verification: the first run should fail completeness, uniqueness, and validity, while freshness should pass because the newest event is nine minutes old. Cleanup is simply removing the temporary script. If this lab were connected to a warehouse, rollback would also include dropping any temporary quality-results table and reverting any quarantined partitions after inspection.

Assessment exercises

A daily revenue table has no null primary keys and no duplicate dates, but the reported total is 18 percent lower than the finance ledger. Which quality dimension is most relevant, and where would you add the check?
A stream processor emits records on time, but the maximum event timestamp is two hours behind. Write the freshness measurement you would store and explain why job success is insufficient.
Choose one check that should block publication and one that should only alert for an orders mart. Justify the difference using consumer impact.
A row-count monitor fires every holiday. Describe how you would distinguish true source loss from expected seasonality without disabling the check.
Design a failed-row sample for an email validity check that helps debugging while reducing exposure of personal data.

Summary

Data quality dimensions turn vague complaints about bad data into specific, testable failure categories. Tests compute measurements and compare them with expectations at defined pipeline points. Observability stores those measurements over time with ownership, lineage, severity, and diagnostic context. The practical engineering skill is choosing the right granularity, threshold, and action for each dataset so that severe defects are blocked, uncertain drift is monitored, and failures can be diagnosed from the recorded evidence.