Open table formats solved a real problem: they let many engines share the same tables on cheap object storage, with transactions, schema evolution and time travel. Apache Iceberg has become a common choice, and the format version 3 specification6 is now supported across much of the ecosystem.
The question I get from data leaders is simple: should we upgrade? The answer depends less on the features than on every engine that touches your tables.
What v3 adds
| Feature | What it does | Who benefits |
|---|---|---|
| Deletion vectors | Compact per-file bitmaps marking deleted rows, replacing positional delete files14 | Tables with frequent updates, deletes and merges |
| Row lineage | Per-row identifiers and last-updated sequence numbers2 | Incremental processing and change data capture |
| Variant type | Native storage for semi-structured data3 | Event, log and API payload data currently stored as strings |
| Geospatial types | Geometry and geography types1 | Logistics, mobility, real estate and mapping workloads |
| Nanosecond timestamps and default values | Finer time precision; defaults for new columns1 | Financial and telemetry data; smoother schema evolution |
Deletion vectors and row lineage are the two I see mattering most. The first makes row-level changes cheaper; the second makes "what changed since last time?" a property of the table rather than a separate pipeline, which simplifies the patterns in change data capture and event streams.
The catch: it's one-way
AWS guidance is explicit that upgrading a table from v2 to v3 can't be reversed through standard operations, and that v2 readers can't read v3 tables. It also notes that the upgrade does not backfill row lineage history2.
Engine support also varies in detail. Amazon Redshift's documentation, for example, lists v3 data types it does not support5. A table that works perfectly in Spark could become unreadable to a BI tool, an ML feature pipeline or a partner's engine the day you upgrade it.
Before upgrading any table, list every engine, service and tool that reads it, including the ones owned by other teams. One unsupported reader turns an upgrade into an outage.
v2 or v3: how I decide
| Stay on v2 for now if | Move to v3 if |
|---|---|
| Any reader lacks confirmed v3 support | All readers support v3 and you've tested them |
| Tables are mostly append-only | Tables see heavy updates, deletes or merges |
| Change tracking already works well | You need incremental processing without extra pipelines |
| Semi-structured data is rare | JSON payloads are common and painful to query |
An upgrade plan
- Inventory readers and writers per table, with their versions and v3 support status.
- Create new tables on v3 where all their consumers support it. New tables carry no compatibility risk.
- Upgrade high-benefit tables first, such as those with heavy merge workloads, after copying and testing a sample in a separate namespace.
- Record format versions in the catalog and your data contracts, so consumers know what to expect.
- Leave stable, append-only tables alone until there is a reason to move them.
Treat it like any platform change: an architecture decision record per domain, a rollback plan that doesn't depend on downgrading, and monitoring for read failures.
Why this matters for AI
Lakehouses increasingly feed AI: training data, feature pipelines and document stores for retrieval. Row lineage helps keep downstream indexes and features in sync with source changes, and variant columns make it easier to keep raw payloads queryable. Both reduce the custom plumbing that tends to break quietly between a data platform and an AI product.
How I can help
I design and review data platforms for scale, from table formats and catalogs to the pipelines that feed analytics and AI. As a fractional CTO I can own the data roadmap, or I can assess an existing platform in technical due diligence. Or analyse your challenge and bring the result to a call.
- 1What's new in Apache Iceberg v3?Google Open Source Blog · Aug 2025
- 2Working with Iceberg table format specification version 3AWS Prescriptive Guidance
- 3
- 4Apache Iceberg v3Starburst
- 5Iceberg v3 features (Amazon Redshift)AWS documentation
- 6Iceberg table specApache Iceberg