Delta Lake Apache Iceberg and Hudi Table Formats

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose and Outcome

Delta Lake, Apache Iceberg, and Apache Hudi solve the same lakehouse problem: object storage is cheap and scalable, but raw Parquet files in a directory are not a table. A data engineering pipeline needs atomic commits, schema evolution, partition changes, deletes, updates, time travel, and predictable query planning. These formats add a transaction and metadata layer above files in S3, ADLS, GCS, or HDFS so Spark, Flink, Trino, Presto, Hive, and other engines can treat file sets as durable tables.

By the end of this lesson, you should be able to explain what a reader actually loads before scanning data, how writers publish a new snapshot or instant, why small files and delete files matter, and how to choose between Delta Lake, Iceberg, and Hudi for streaming ingestion, warehouse-style analytics, and mutable operational datasets.

How the Formats Work Internally

A plain Parquet dataset usually relies on directory names such as dt=2026-09-07/region=west. If a job fails halfway through writing files, readers may see partial data. If a schema changes, old files and new files can disagree. If rows are deleted, the system has no table-level place to record which rows are no longer valid. Table formats fix this by separating data files from table metadata.

Delta Lake stores an ordered transaction log under _delta_log. Each JSON commit records actions such as add, remove, metaData, and protocol. Periodic Parquet checkpoints compact many JSON log entries so readers do not replay the full table history. A writer creates new data files, then atomically publishes a new log version. Readers pick a version, replay actions up to that version, and scan only active files.

Apache Iceberg uses metadata files rather than a linear JSON log as the central abstraction. A catalog pointer identifies the current table metadata file. That file points to snapshots. A snapshot points to manifest lists, and manifests list data files and delete files with partition and column statistics. A commit succeeds when the catalog pointer is atomically swapped from the old metadata file to the new one. This design makes hidden partitioning, snapshot references, and multi-engine planning central features.

Apache Hudi organizes changes on a timeline of instants such as requested, inflight, and completed commits. Hudi tables are commonly copy-on-write, where updates rewrite Parquet files, or merge-on-read, where updates are appended to log files and compacted later. Hudi also has indexing options to find the file group containing a record key, which is why it is often used for upserts from change data capture streams.

API and Configuration Anatomy

The table path or catalog name identifies the table. The schema defines columns and types. The partition specification controls file layout or logical partition transforms. Writer options define merge keys, commit behavior, target file sizes, and compaction or clustering policies. Reader options select the current version, a timestamp, or a named branch where supported.

-- Delta Lake
CREATE TABLE sales_delta (
  order_id BIGINT,
  customer_id BIGINT,
  amount DECIMAL(12,2),
  order_ts TIMESTAMP
) USING DELTA
PARTITIONED BY (date(order_ts));

-- Iceberg
CREATE TABLE lake.sales_iceberg (
  order_id BIGINT,
  customer_id BIGINT,
  amount DECIMAL(12,2),
  order_ts TIMESTAMP
) USING ICEBERG
PARTITIONED BY (days(order_ts));

-- Hudi
CREATE TABLE sales_hudi (
  order_id BIGINT,
  customer_id BIGINT,
  amount DECIMAL(12,2),
  order_ts TIMESTAMP
) USING HUDI
TBLPROPERTIES (
  primaryKey = 'order_id',
  preCombineField = 'order_ts',
  type = 'cow'
);

The important distinction is not the spelling of CREATE TABLE. Delta emphasizes an append-only transaction log. Iceberg emphasizes snapshot metadata trees and catalog commits. Hudi emphasizes record keys, file groups, and a timeline that supports incremental consumption.

Worked Example 1: Atomic Visibility

This Python model shows the core idea behind a table format. Data files can be written first, but readers only see files referenced by the committed metadata version.

from dataclasses import dataclass

@dataclass(frozen=True)
class Commit:
    version: int
    active_files: tuple[str, ...]

commits = [Commit(0, ("part-000.parquet",))]
staged_file = "part-001.parquet"

# A failed writer leaves staged_file on storage but never commits version 1.
reader_snapshot = commits[-1]
print(reader_snapshot.version, list(reader_snapshot.active_files))

The output is version 0 with only part-000.parquet. The staged file may exist in storage, but a correct Delta, Iceberg, or Hudi reader ignores it because table metadata has not made it active.

Worked Example 2: Snapshot Replacement

Deletes and updates usually create new files and mark old files as removed, or create delete records that are applied at read time. The old snapshot remains available for time travel until retention cleanup removes it.

snapshots = {
    10: {"A.parquet", "B.parquet"},
    11: {"A.parquet", "C.parquet"},
}
removed = snapshots[10] - snapshots[11]
added = snapshots[11] - snapshots[10]
print("removed", sorted(removed))
print("added", sorted(added))

The deterministic output is removed ['B.parquet'] and added ['C.parquet']. A query at snapshot 10 still reads B.parquet. A current query reads C.parquet instead. Iceberg may also use position deletes or equality deletes; Delta commonly records remove actions; Hudi copy-on-write rewrites file groups while merge-on-read can defer merging until compaction or query time.

Worked Example 3: Incremental Consumption

Hudi and Delta change data feed features, and Iceberg snapshot scans, let downstream jobs process only new committed changes. The pipeline must store its last processed instant or snapshot, not just a wall-clock timestamp.

timeline = [
    {"instant": "001", "rows": 100, "operation": "insert"},
    {"instant": "002", "rows": 12, "operation": "upsert"},
    {"instant": "003", "rows": 4, "operation": "delete"},
]
last_processed = "001"
changes =  > last_processed]
print(sum(c["rows"] for c in changes),  for c in changes])

The output is 16 ['upsert', 'delete']. This is the lakehouse version of a checkpoint. If the consumer records 003 only after its own sink commit succeeds, it can recover without skipping committed table changes.

Design Choices and Trade-offs

Choose Delta Lake when your platform is Spark-centered or Databricks-centered, you want a straightforward transaction log model, and you value mature commands such as optimize, vacuum, merge, and change data feed. Its operational model is easy to reason about, but protocol features and engine support must be checked carefully when several query engines share a table.

Choose Iceberg when you need open multi-engine interoperability, hidden partition evolution, strong query planning metadata, and catalog-controlled commits. Iceberg avoids exposing physical partition directories as the table contract. That helps when a table starts partitioned by day and later needs hour or bucket transforms. The trade-off is that catalog configuration and metadata maintenance become part of the platform design.

Choose Hudi when upserts, deletes, and incremental pulls from mutable source systems dominate the workload. Hudi record keys, precombine fields, indexes, and merge-on-read tables are built for change streams. The trade-off is more tuning: compaction, clustering, indexing, and payload semantics can strongly affect latency and cost.

Failure Modes and Troubleshooting

Symptom: queries miss recently written data. Likely cause: the writer produced files but did not publish a successful commit, or the reader is pinned to an older version. Check the transaction log, Iceberg current metadata pointer, or Hudi completed instant timeline. Correct by retrying the failed job with idempotent input boundaries and verifying that the new version or instant is completed.

Symptom: queries are slow even though only a small date range is requested. Likely cause: partition pruning or statistics pruning is ineffective, often because the filter does not match the partition transform or because many small files were created. Inspect file counts, manifests or log metadata, partition summaries, and query plans. Correct by compacting or optimizing files and aligning filters with the partition specification.

Symptom: concurrent writers fail with commit conflicts. Likely cause: two jobs attempted incompatible changes to the same table state. Diagnose by reading the failed commit error and identifying whether the jobs touched the same partitions, files, or metadata. Correct by narrowing write predicates, serializing maintenance jobs, or using a catalog and engine combination with the required optimistic concurrency guarantees.

Symptom: old snapshots cannot be queried. Likely cause: retention cleanup removed old data files or metadata. Check vacuum, expire snapshots, or cleaner configuration. Correct by setting retention according to audit and backfill requirements, then restoring from object storage versioning or backup if files were already deleted.

Security, Performance, and Reliability

Access control has to cover both metadata and data files. A user who can read old metadata paths may discover historical file locations, and a user who can bypass the catalog may avoid table-level governance. In regulated environments, put authorization at the catalog or warehouse layer and restrict direct object-store access.

Performance depends on file size, metadata size, statistics quality, and delete strategy. Too many tiny Parquet files increase planning and listing overhead. Too many equality or position deletes can make reads CPU-heavy until compaction rewrites clean files. Reliability depends on atomic commit support in the chosen catalog or storage layer and on maintenance jobs that do not delete files still needed by readers.

Hands-on Lab: Compare Metadata Behavior

Prerequisites: a local Spark environment with Delta, Iceberg, or Hudi packages, or a managed notebook that supports at least one of them; permission to create and delete test tables; and an empty scratch location.

  1. Create a small table with columns id, status, and updated_at.
  2. Insert three rows and query the table. Record the current version, snapshot id, or Hudi instant.
  3. Update one row and delete one row using the format’s supported SQL or DataFrame API.
  4. List the table metadata directory or catalog history. For Delta, inspect _delta_log. For Iceberg, inspect snapshots and manifests through system tables. For Hudi, inspect the timeline and file groups.
  5. Run a current query and, if supported in your environment, a time-travel query against the original version.
  6. Run a compaction, optimize, clustering, or snapshot-expiration command only on the scratch table.

Verification: the current query should show the updated row, omit the deleted row, and retain the untouched row. The history view should show at least two committed table states. A time-travel query should match the first insert result if retention has not removed the required files.

Cleanup: drop the scratch table, remove the scratch object-store prefix or local directory, and delete any catalog entry created for the lab. Do not run vacuum or snapshot expiration on shared tables while testing.

Assessment Exercises

  1. A Spark job writes Parquet files into a table location and crashes before committing. Explain why a table-format-aware reader should not return those rows.
  2. Design a partition evolution plan for an events table that starts with daily queries but later needs hourly pruning. Which format features make the change less disruptive?
  3. A Hudi merge-on-read table has low write latency but slow analytical queries. Identify two diagnostics and one corrective maintenance action.
  4. Compare rewriting files for updates with applying delete files at read time. When is each approach likely to be cheaper?
  5. Explain why an incremental downstream job should checkpoint a table snapshot or instant instead of only checkpointing the maximum event timestamp.

Summary

Delta Lake, Apache Iceberg, and Hudi are not storage engines; they are table formats that make object-store files behave like transactional analytical tables. Delta’s log, Iceberg’s snapshot and manifest tree, and Hudi’s timeline and file groups each solve atomic visibility, evolution, and mutation differently. Strong data engineering practice means choosing the format whose metadata model matches the workload, then operating compaction, retention, catalog permissions, and incremental checkpoints with the same care as the ingestion code.