A few years ago the lakehouse argument was about table formats. That argument is largely settled: Apache Iceberg is widely supported, and I've covered its latest version in Apache Iceberg v3. The files sit in your own object storage in an open format any engine can read.
The quieter question is who every engine asks before reading those files. That is the catalog, and it has become the most strategic choice in a modern data platform.
Why the catalog holds the power
In an Iceberg lakehouse, the catalog tracks which tables exist, their current metadata and schema, and increasingly who may access them. Modern catalogs go further and issue engines short-lived storage credentials scoped to the tables they're allowed to read, a pattern called credential vending. Databricks' documentation, for example, describes Unity Catalog granting short-lived credentials to external systems that connect through its REST API or the Iceberg REST catalog interface, subject to explicit privileges and metastore settings4.
That is good security design. It also means the catalog is where permissions, lineage, audit and access for every engine live. Changing the catalog later is far harder than changing an engine.
Open files with a closed catalog is a more polite form of lock-in, but it's still lock-in. Evaluate catalogs as carefully as you once evaluated databases.
The open REST interface
The Iceberg REST catalog API is what makes multi-engine lakehouses practical: any engine that speaks it can work with any catalog that implements it. Apache Polaris is an open-source catalog built around that API, described by the Apache Software Foundation as a fully featured catalog for Iceberg that enables interoperability across engines such as Spark, Flink and Trino1.
Polaris entered the Apache incubator in 2024 after being donated by Snowflake and Dremio, and the foundation announced it as a top-level project in March 202612. Commentary on the graduation noted multiple releases and a broad contributor base2, and the 1.4 release added multi-tenancy and deployment improvements3. Top-level status means independent Apache governance, which matters if neutrality is part of why you chose an open catalog.
Unity Catalog takes a different route: a catalog with its own API that also exposes Iceberg REST access for external engines, alongside Delta support45. Check the current status of each capability, since some external access features have been labelled preview.
Questions I ask when choosing
| Question | Why it matters |
|---|---|
| Which engines must read and write, today and in three years? | Every engine needs a supported path through the catalog, including write access |
| Where are permissions defined, and can they be exported? | Governance that can't leave the catalog is the deepest form of lock-in |
| Does it vend scoped, short-lived credentials? | Avoids engines holding broad storage keys |
| Who governs the project or product? | Foundation governance and vendor roadmaps carry different risks |
| Can it be self-hosted in-region? | Residency rules, as in Saudi PDPL and UAE health data, may require it |
| How are lineage and audit exposed? | Regulators and AI governance increasingly ask where data came from |
Patterns that keep options open
- Standardise on the Iceberg REST interface for engine access, even if your catalog also offers proprietary APIs. That keeps engines swappable.
- Keep governance definitions as code: roles, grants and policies in version control, applied to the catalog, so they can be reapplied elsewhere.
- Use one catalog per governance domain, not one per engine. Several catalogs over the same tables invite inconsistency.
- Pair the catalog with data contracts, so table semantics and ownership travel with the data rather than living only in catalog metadata.
- Test an exit. Once a year, prove that a second engine, or a second catalog, can read critical tables. It's cheap insurance.
The AI angle
AI systems are now among the biggest consumers of lakehouse data: feature pipelines, training sets, retrieval indexes and agents querying tables directly. A catalog that vends scoped credentials and records access gives you one place to control what each AI workload may read, which supports the evidence expected in AI governance frameworks. Without it, every AI project tends to get its own copy of the data, and governance dissolves.
How I can help
I design lakehouse and data platform architectures that stay open where it matters, from table formats and catalogs to governance and AI access. As a fractional CTO I can own the data platform roadmap, or review an existing one in technical due diligence. Or analyse your challenge and bring the result to a call.
- 1The Apache Software Foundation graduates two open source projects from incubatorThe Apache Software Foundation · Mar 2026
- 2
- 3
- 4Unity Catalog credential vending for external system accessDatabricks documentation
- 5