- Quality, production and supplier data joinable at batch and lot level across plants
- Each plant's feed governed by a published contract with an owner
- One agreed definition for each quality metric, used by dashboards and analysts alike
- Breaking changes caught at the plant boundary instead of in month-end reports
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A manufacturer with several plants had good systems and poor connections. Quality inspection results lived in a central quality system. Production parameters lived in each plant's execution and historian stack, configured differently at every site. Supplier certificates and incoming inspection data arrived as spreadsheets and portal exports. When a customer complaint came in, tracing it back to a material lot, a line, a shift and a process setting took days of manual joining, and by then more product had shipped.
Previous attempts at a central warehouse had stalled. Every plant change broke a pipeline, and every report defined "first-pass yield" slightly differently.
What had to be true
- Joinable at the right grain: batch, lot, line and time, across quality, production and supplier data.
- Plants accountable for their feeds, with a clear owner and a written promise for each.
- Open storage, so different teams and engines could use the same tables.
- One definition per metric, shared by dashboards, analysts and any future AI use.
- No rip and replace of plant systems.
ISA-95 framed the boundaries. It addresses the interface between enterprise business systems and manufacturing operations and control1, which is exactly where the lakehouse sat: reading from plant-level systems, serving enterprise-level analysis, owning neither.
Architecture
Apache Iceberg tracks table state through metadata files replaced by atomic swaps, with snapshots and schema evolution defined in the specification2. That gave reproducible reads for investigations and safe schema changes when a plant added a parameter. See Apache Iceberg v3 for the lakehouse and lakehouse catalog strategy.
The data flow, end to end
- CaptureChange data capture or scheduled extracts from each plant system
- CheckValidate against the feed's contract; quarantine and notify the owner on failure
- LandAppend to raw Iceberg tables with source and contract version recorded
- ConformMap plant codes to shared reference data; build lot genealogy
- DefineCompute quality metrics once, in the semantic layer
- ActDashboards, alerts and investigations read governed metrics only
Capture patterns are covered in change data capture and event streams.
The hard part: getting plants to own a promise
The technology was well understood. The hard part was organisational: plants had never been asked to promise anything about their data, and a central team had always absorbed the breakage. Contracts changed that, but only once they were written in terms plants recognised.
We used the Open Data Contract Standard, a YAML-based, platform-neutral format for agreements between data producers and consumers3; version 3 organises a contract into sections that include schema, data quality, service levels and team4. In practice:
- Contracts started small: schema, owner, freshness and a handful of rules the plant itself agreed were reasonable.
- A failed check went to the plant's named owner, not to the central team's backlog.
- Breaking changes required a new contract version and a notice period, agreed in a short change process.
- Plant-specific codes stayed in the feed; mapping to shared reference data was a central, versioned step.
A data contract is an organisational tool that happens to be written in YAML. Its value is the conversation it forces between producer and consumer.
More in the data contracts playbook.
How the engagement ran
- Agree the contract
- Fix failed checks at source
- Announce changes
- Run checks and quarantine
- Conform and join
- Publish curated tables
- Define metrics once
- Review alerts
- Drive supplier actions
- Walk an investigation end to end
- Source inventory
- Metric disagreements
- First contracts
- Iceberg tables and catalog
- Lot genealogy
- Semantic layer
- Agreed definitions
- Dashboards and alerts
- Further plants
- Supplier feeds
- Change process
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Apache Iceberg tables | Proprietary warehouse storage | Multiple engines on one copy; no lock-in |
| Contracts at the plant boundary | Central team fixes breakages | Fixes happen where the change was made |
| Metrics in a semantic layer | Definitions in each dashboard | One answer to "what is our yield?" |
| Keep plant codes, map centrally | Force plants to standardise first | Delivery without waiting on every plant |
| Pilot one plant end to end | All plants at once | Contract and genealogy lessons learned once |
Why metrics belong in a semantic layer, especially if AI will query them later, is covered in text-to-SQL and the semantic layer.
What I'd do again, and differently
Again: start by tracing one real complaint end to end, and make plants owners rather than suppliers of data. Differently: agree metric definitions with quality leads before building dashboards, not alongside them. Every disagreement found late cost a rebuild.
If you're facing something similar, the fractional CTO page describes how I run these engagements, or analyse your challenge and bring the result to a call.
- 1ISA-95 Standard: Enterprise-Control System IntegrationInternational Society of Automation
- 2Iceberg Table SpecApache Iceberg
- 3Open Data Contract Standard (ODCS)Bitol, Linux Foundation (GitHub)
- 4