Pharma medical affairs, payers, health systems and regulators all want the same thing from real-world data: evidence about what treatments actually do outside trials. The data, mostly claims and electronic health records, is plentiful. The hard part is design. Every observational study is an attempt to estimate a counterfactual, what would have happened to these patients without the exposure or with a different one, from data that only records what did happen1.
The OHDSI community implements five main designs for this as open-source R packages in its Methods Library, all running on OMOP data13. Knowing what each one assumes is the difference between evidence and a confident-looking chart.
The five designs at a glance
| Design | Compares | Main strength | Main vulnerability |
|---|---|---|---|
| Cohort method (new user) | New users of target vs new users of comparator | Emulates a trial; rich diagnostics | Unmeasured confounding between groups |
| Self-controlled cohort | Exposed time vs time just before exposure | No between-person confounding | Baseline risk differing between periods |
| Case-control | Exposure in people with vs without the outcome | Efficient for rare outcomes | Choice of controls |
| Case-crossover | Exposure at outcome time vs an earlier control time | Cases are their own controls | Trends in exposure over time |
| Self-controlled case series | Outcome rate in exposed vs all unexposed time | No between-person confounding | Time-varying confounding; outcomes that end observation |
All rows are drawn from the descriptions in The Book of OHDSI1.
Cohort method: emulating a trial
The new-user cohort design compares people who start the target treatment with people who start a comparator, and follows them for a defined time-at-risk. It is defined by five choices: target cohort, comparator cohort, outcome cohort, time-at-risk and model1.
The central problem is that patients on different treatments differ. OHDSI's preferred adjustment is a propensity score fitted with large-scale regularised regression over many generic characteristics, typically 10,000 to 100,000 of them, rather than a hand-picked list1. The design comes with diagnostics: preference score distributions to check the groups overlap, and covariate balance after adjustment, where some guidelines recommend standardised differences below 0.114.
The book is clear that this is inherently a comparative design. Comparing a treatment with no treatment is hard because a comparable unexposed group is difficult to define; an active comparator for the same indication is preferred1.
Self-controlled designs: people as their own controls
Three designs avoid between-person comparison altogether. The self-controlled cohort compares outcome rates during exposure with the time just before. The case-crossover design asks whether exposure was more common just before the outcome than at an earlier control time; because the outcome always comes later, it can be biased by exposure trends, which the case-time-control variant addresses. The self-controlled case series compares the outcome rate during exposed time with all unexposed time, before, between and after exposures1.
These designs are robust to confounding by stable characteristics such as genetics or chronic conditions, but vulnerable to things that change over time, such as age, season or disease progression1. The self-controlled case series also assumes the end of observation is independent of the outcome, which can fail for fatal outcomes such as stroke; an extension exists to correct for that15.
No design is assumption-free. Choose the one whose weaknesses matter least for your question, then test it on questions where you already know the answer.
Case-control: efficient, but fragile
Case-control studies compare exposure in people with the outcome against people without it, often matching on age and sex and sometimes nesting within a subgroup such as patients with the relevant indication1. They are efficient for rare outcomes, but results depend heavily on how controls are selected.
How I'd choose and validate
- Write the question precisely as cohorts and time windows, as in OMOP cohort definitions.
- Default to the new-user cohort design with an active comparator for comparative effectiveness, because its diagnostics make problems visible.
- Use self-controlled designs for acute outcomes after transient exposures, such as vaccine or short-course drug safety, where timing is sharp.
- Run negative controls. Outcomes believed not to be caused by the exposure reveal residual bias; the book's method validity chapter covers this and empirical calibration2.
- Replicate across databases through the federated network approach, after data quality checks at each site.
- Keep estimation and prediction separate. Causal questions and prediction models need different designs and different validation.
How I can help
I design OMOP platforms and analytics pipelines that make real-world evidence studies reproducible, diagnosable and fast to run across sites. See healthcare data architecture, or analyse your challenge and bring the result to a call.
- 1
- 2
- 3
- 4
- 5