Reference · print me
Anatomy of an Iceberg table
The metadata tree, the commit protocol, the transforms, and the version matrix — on one page.
1.The tree, top to bottom
2.Field names you will meet
| File | Fields |
|---|---|
| Table metadata | format-version · table-uuid · location · last-sequence-number · last-updated-ms · last-column-id · schemas · current-schema-id · partition-specs · default-spec-id · sort-orders · properties · snapshots · current-snapshot-id |
| Snapshot | snapshot-id · parent-snapshot-id · sequence-number · timestamp-ms · manifest-list · summary (must contain operation) · schema-id · first-row-id (v3) |
| Manifest list entry | manifest_path · manifest_length · partition_spec_id · content (0 = data, 1 = deletes) · sequence_number · min_sequence_number · added_snapshot_id · added_files_count · existing_files_count · deleted_files_count · partitions |
| Manifest entry | status (0 existing / 1 added / 2 deleted) · snapshot_id · sequence_number · file_sequence_number · data_file {path, partition tuple, record count, column bounds} |
| Deletion vector (v3) | Tracked in a delete manifest by file_path · content_offset · content_size_in_bytes. Stored as a deletion-vector-v1 Puffin blob. |
3.The commit protocol
- Read current table metadata; note the base version.
- Write new data files. (Invisible to the table — nothing points at them.)
- Write a manifest for them.
- Write a new manifest list: the new manifest plus references to the unchanged ones.
- Write a new metadata JSON with the new snapshot, optimistically assigned the next sequence number.
- Ask the catalog to swap the pointer, check-and-put against the base version.
- Rejected? Refresh, re-check the commit's assumptions against the new state, re-apply the actions, and go to 6. The data files from step 2 are reused.
Filesystem "catalogs" are deprecated
The spec calls the file-system commit scheme unsafe in object stores and local file systems; it is being removed in v4. A real catalog with a check-and-put is not optional on S3.
4.Partition transforms
| Transform | Result | Applies to |
|---|---|---|
identity | the value | any primitive except geometry / geography |
bucket[N] | int | numeric, temporal, string, uuid, fixed, binary |
truncate[W] | source type | int, long, decimal, string, binary |
year / month / day / hour | int from the epoch | date (not hour), timestamp, timestamptz, and the _ns variants |
void | always null | any — used to retire a v1 partition field |
bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N, seeded with 0. Multi-argument transforms are v3+.
5.What is metadata-only, and what rewrites data
Free — metadata only
- Add, drop, rename, reorder a column
- Widen a type (see below)
- Change the partition spec
- Roll back to a snapshot
- Tag or branch a snapshot
Moves bytes
- Compaction / re-sort
- Copy-on-write deletes and updates
- Backfilling old data into a new layout
- Rewriting manifests
Type promotion, and only these: int→long; float→double; decimal(P,S)→decimal(P',S) when P' > P; and in v3+ date→timestamp/timestamp_ns and unknown→anything. Promotion to timestamptz is not allowed.
6.Format version matrix
| Version | Adds | Status |
|---|---|---|
| v1 | Analytic tables over immutable files: snapshots, manifests, hidden partitioning, schema evolution | Adopted |
| v2 | Row-level deletes: position delete files, equality delete files, sequence numbers | Adopted |
| v3 | Deletion vectors · row lineage · variant · geometry/geography · timestamp_ns · unknown · default values · multi-argument transforms · table encryption keys. Position delete files deprecated. | Adopted |
| v4 | Metadata restructuring; relative locations in metadata fields | Not adopted — under active development |
Compatibility is one-way: a v3 reader reads v2 tables; a v2 reader cannot read v3 tables.
7.Delete semantics at a glance
| Copy-on-write | Merge-on-read | |
|---|---|---|
| On delete | Rewrite the affected data files | Write a delete file / deletion vector |
| Write cost | High | Low |
| Read cost | None | Apply deletes at scan time |
| Best for | Rare deletes, read-heavy tables | Frequent deletes, streaming upserts |
A delete applies to a data file only when the data file's sequence number is ≤ the delete's, and (for a DV or position delete) the paths match. In v3 there is at most one deletion vector per data file per snapshot.