- Studies specified once and executed at every data partner without moving patient-level data
- Bias measured with negative controls before results were unblinded
- Each result reproducible from a pinned study package, data version and environment
- A partner onboarding path that made new data sources usable for existing studies
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A life sciences evidence team answered questions about treatment patterns, comparative safety and outcomes using data licensed from several external partners: claims databases, EHR-derived datasets and a disease registry. Each study started the same way: negotiate an extract, receive files in the partner's format, spend months cleaning and harmonising, then write analysis code that only worked on that one extract. When a reviewer later asked a question, re-creating the original analysis was an archaeology project.
The partners were also reluctant to send patient-level data at all. The team needed a way to run studies where the data lived, with results the organisation would be comfortable defending to a regulator.
What had to be true
- One common model at every partner, so study code runs unchanged.
- Design before data: protocols, cohorts and analyses pre-specified and locked before results are seen.
- Bias measured, not assumed, using controls with a known answer.
- Exact reproducibility of any result, months or years later.
- Data stays with the partner; only reviewed aggregates come back.
The FDA's framework for its real-world evidence programme frames the questions such evidence will face, including whether the data are fit for use and whether the study design can provide adequate evidence1. Those questions shaped the architecture from the start.
Architecture
Analytics used OHDSI's HADES packages, open-source R tools for characterisation, population-level effect estimation and patient-level prediction against the OMOP CDM2. Partner conversions followed the patterns in FHIR to OMOP architecture and OMOP vocabularies where partners started from scratch.
The data flow, end to end
- QuestionResearch question and protocol written; estimand, cohorts and comparators fixed
- FeasibilityAggregate counts from each partner confirm the question is answerable there
- PackageCohorts, analyses and negative controls bundled as one specification with a pinned environment
- ExecuteEach partner runs the package locally against its current OMOP release
- DiagnoseBalance, power and negative control results reviewed before any effect estimate is unblinded
- SynthesiseCalibrated estimates combined across partners; everything archived with versions
Cohort practice is covered in OMOP cohort definitions, and study design choices in real-world evidence study designs on OMOP.
The hard part: proving the design is not fooling you
Observational studies can produce confident, wrong answers. OHDSI's approach to method validity uses negative controls, exposure and outcome pairs believed to have no causal effect, to measure residual systematic error, and empirical calibration to adjust confidence intervals and p-values for it3. Making that routine required:
- A curated negative control list per study, reviewed by clinical scientists rather than generated and forgotten.
- A rule that effect estimates stay blinded until diagnostics (covariate balance, equipoise, negative control distribution) pass.
- Diagnostics reported per partner, because bias that appears in one data source and not another is itself a finding.
The most valuable output of a negative control analysis is permission to say "this design should not be trusted here". Building that into the process is what makes the other results credible.
How the engagement ran
- Locks protocol
- Builds study package
- Reviews diagnostics
- Maintains OMOP release
- Runs package locally
- Releases aggregates
- Package and environment
- Data and vocabulary versions
- Results and diagnostics
Execution used Strategus, which runs HADES modules from a JSON analysis specification and restores a pinned R environment, separating study design from site-specific execution settings4.
- Partner OMOP readiness
- Data cards and quality baselines
- Governance with partners
- One question, two partners
- Full diagnostics
- Archive format
- Reusable cohorts
- Negative control sets
- Package templates
- More partners
- Study intake process
- Hand-off and training
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| OMOP at every partner | Harmonise extracts centrally | Study code runs unchanged; data stays with the partner |
| Locked, packaged studies | Analysts iterate on live data | Prevents result-driven design changes; enables exact re-runs |
| Negative controls in every estimation study | Rely on adjustment alone | Measures residual bias instead of assuming it away |
| Blind estimates until diagnostics pass | Review everything together | Keeps the decision to trust a design independent of its answer |
| Open-source OHDSI tooling | Proprietary analytics platform | Transparent methods reviewers can inspect and partners already run |
What I'd do again, and differently
Again: treat the archived study package as the deliverable, not the slide with the hazard ratio. Differently: invest earlier in partner data cards. Knowing each source's coverage, lag and coding habits up front would have saved feasibility rounds on questions some partners could never answer.
If you are building evidence capability across data partners, the healthcare data architecture page describes how I run these engagements, or analyse your challenge and bring the result to a call.
- 1Framework for FDA's Real-World Evidence ProgramUS Food and Drug Administration · Dec 2018
- 2
- 3
- 4StrategusOHDSI (GitHub)