OMOP data quality in practice: conformance, completeness and plausibility

A shared vocabulary for "quality"

The framework I use comes from Kahn and colleagues, who published a harmonised terminology for EHR data quality in 2016². It has three categories:

Each category can be checked by verification, against internal rules and expectations, or validation, against an external benchmark such as other sites or published rates¹². The value of the framework is less in the categories than in the shared language: clinicians, analysts and engineers stop arguing about what "bad data" means.

Layer 1: characterise the data with ACHILLES

ACHILLES pre-computes a large set of characterisation analyses over an OMOP CDM, such as distributions of drug supply days by concept, and presents them visually¹. I use it first because it shows the shape of the data before anyone writes a rule.

Layer 2: rule-based checks with the Data Quality Dashboard

The Data Quality Dashboard runs rule-based checks against the CDM and reports, for each one, the fraction of rows that violate it, a threshold and a pass or fail status. Checks are organised by the Kahn categories and by table, field and concept level¹³.

Two practical notes. Thresholds are a decision, not a default, so agree them with the people who will use the data. And a failed check can come from the conversion or from the source system itself; the Book of OHDSI is explicit that dashboard findings need that diagnosis before anyone fixes anything¹.

Layer 3: unit tests for the ETL

Dashboards find problems after the fact. ETL unit tests stop them getting in. The Book of OHDSI recommends a suite of small tests, each creating a fake source record and asserting what the CDM should contain, typically hundreds of them, re-run whenever the ETL changes¹.

This is ordinary software engineering applied to clinical data, and it is the layer most often skipped. In my view it is the most valuable, because mapping mistakes are the ones you can actually prevent.

Layer 4: study-specific checks

General checks say the data is broadly usable. A specific study needs more: confirm the concepts in your cohort definitions appear at plausible rates over time, check that the key exposure fields are populated, and compare cohort prevalence with clinical expectations¹. The same logic applies before training a clinical model: check the data for the question you are asking, not just in general.

Making it operational

1. Run ACHILLES and the Dashboard on every refresh, not once at go-live, and keep the history so trends are visible.
2. Gate releases on agreed thresholds. A refresh that fails critical checks doesn't reach analysts.
3. Keep ETL unit tests in CI alongside the conversion code.
4. Publish a data quality report with each release, so users know the known limitations.
5. Track mapping coverage: how many source codes map to standard concepts, and which unmapped codes matter most.

This is the quality layer on top of the pipeline I describe in FHIR to OMOP: a practical architecture.

How I can help

I design healthcare data platforms where quality is measured, gated and reported, from ETL tests to dashboards. See healthcare data architecture, or analyse your challenge and bring the result to a call.