- A sandbox plan with specific questions, success criteria and exit conditions
- Evidence from live wards gathered first in silent mode, then with clinicians
- Performance tracked by site and patient group, not only overall
- Documentation built during the sandbox rather than reconstructed afterwards
- A clear understanding that sandbox exit reports support, but do not replace, conformity assessment
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A European medtech start-up had developed a model that estimates the risk that a patient on a general hospital ward will deteriorate in the next day, using vital signs, laboratory results and nursing observations. On historical data from two partner hospitals it performed well. The founders knew that was the easy part. Clinical AI often fails in the move from historical data to live wards: data arrives late or differently, staff respond to alerts in ways nobody predicted, and performance varies between sites and patient groups.
As software intended to support clinical decisions, the product was likely to be regulated as a medical device and treated as high-risk under the EU AI Act. The team needed evidence that would satisfy both regimes, and had no experience of what supervisors would expect.
Why now. On 11 November 2026, the Commission published Implementing Regulation (EU) 2026/2487, setting common rules for establishing and running AI regulatory sandboxes under the AI Act12. Commentary was quick to point out the limits: sandbox participation produces documentation and an exit report, not compliance clearance, and liability stays with the provider34. That suited the start-up: it wanted structured supervision and early feedback, not a shortcut.
What had to be true
- Clear questions: the sandbox plan states exactly what will be tested and what success looks like.
- Patient safety first: no clinician acts on the model until it has been validated on live data.
- Evidence that travels: results usable later for the AI Act and medical device assessments.
- Performance by group: results reported by site, ward type and patient group.
- Reproducible: any result can be regenerated from versioned data, features and model.
- No false comfort: everyone understands what the sandbox does and does not provide.
Architecture
Hospital data arrived through FHIR feeds into a feature pipeline where every feature had a versioned definition, so a result could always be traced to the exact data and code that produced it. The scoring service ran in two modes: silent, where scores were stored but not shown, and live, where alerts reached ward staff. See healthcare data architecture.
The data flow, end to end
- IngestObservations and results arrive from the hospital in near real time
- CheckData quality rules flag late, missing or implausible values
- ScoreThe model estimates deterioration risk for each patient
- AlertIn live mode, high-risk patients prompt a structured clinical review
- CompareScores are compared with actual outcomes, by site and patient group
- ReportMonthly evidence packs go to the supervisor and clinical leads
The hard part: evidence a supervisor can trust
Designing the sandbox plan. The start-up's first draft asked to "test the model in hospitals". The final plan, agreed with the supervising authority, asked specific questions: does performance on live data match historical results at each site; is the model well calibrated for older patients and patients with chronic conditions; do nurses act on alerts within the expected time; and does alert volume stay manageable on busy wards. Each question had a success criterion and an agreed response if it failed.
Making results reproducible and generalisable. Supervisors and later assessors would ask whether results could be repeated and whether they would hold elsewhere. The team borrowed the vocabulary of the observational research community: a result should be repeatable on the same data, reproducible by someone else, and generalisable to different data5. Versioned data and features handled the first two; adding a third hospital partway through tested the last.

Silent mode before live mode. For the first phase the model scored real patients but showed nothing to clinicians. That exposed data timing problems at one site that would have made live alerts arrive too late to help, and a calibration gap in one patient group that was fixed before any nurse saw a score.
A sandbox is valuable for what you learn with the supervisor watching, not for the badge. Go in with sharp questions, and come out with evidence you would have needed anyway.
How it ran
- Sandbox plan
- Pipeline and monitoring
- Evidence packs
- Data access and governance
- Clinical review process
- Nurse feedback
- Agree questions
- Review progress
- Exit report
- Specific questions
- Success criteria
- Exit conditions
- Live scoring, no alerts
- Data timing fixes
- Calibration by group
- Alerts on selected wards
- Structured reviews
- Alert volume tracking
- Evidence pack
- Exit report
- Assessment plan
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Enter a sandbox with specific questions | Treat it as a general pilot | Clear questions make supervisor feedback useful |
| Silent mode before live alerts | Go live on validated history | Live data behaves differently; patients stay safe |
| Report by site and patient group | One overall score | Averages hide where the model fails |
| Versioned features and data | Ad hoc extracts | Results can be reproduced for assessors |
| Add a third hospital mid-sandbox | Stay with two partners | Tests whether results generalise |
| Plan conformity assessment in parallel | Rely on the exit report | The sandbox does not grant compliance |
What I'd do again, and differently
Again: write the sandbox questions with clinicians first, then take them to the supervisor; it made every later conversation concrete. Differently: plan alert workflow with nursing leads earlier; the hardest live-mode issues were about who responds and when, not about the model. Related: evaluation that predicts production, the FHIR and OMOP evidence platform case study, the 13 November tech news and the Europe page.
If you are building clinical or other high-risk AI for Europe and want evidence that stands up, see my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Commission Implementing Regulation (EU) 2026/2487EUR-Lex · 11 Nov 2026
- 2Commission lays down common rules for AI regulatory sandboxesEU Law Live · Nov 2026
- 3Sandboxes Without Safe Harbor: EU AI Sandbox Rules and Security TestingCloud Security Alliance · 12 Nov 2026
- 4
- 5The Book of OHDSI: Evidence QualityOHDSI · 2021