- Identifiable patient data kept in-country; the model sees de-identified text only
- Red-flag symptoms escalated by rules before any model output
- Every AI suggestion reviewed by a clinician before it affects care
- A data protection impact assessment covering the AI use, kept current with each release
- English and Swahili behaviour tested on every change
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A telehealth provider in Kenya offered consultations by app, WhatsApp and SMS, backed by a network of clinicians and partner pharmacies. Demand peaked in the evenings, when patients waited longest for a first response. The team wanted an AI assistant to take a structured symptom history in English or Swahili, summarise it and suggest who should see the patient, so clinicians could start from a clear picture.
Why now. Kenya's Data Protection Act treats health data as sensitive personal data and allows transfers abroad only with appropriate safeguards1. The Office of the Data Protection Commissioner (ODPC) consulted in 2026 on draft guidance on AI2, and commentary on the draft says uses such as health diagnostics and clinical decision support would need a data protection impact assessment3. There were also public calls for stronger protection of data sent to offshore AI providers4, and an October 2026 report described African regulators, Kenya's among them, using data protection law to police AI while dedicated AI bills stall5. A dedicated AI law may or may not arrive soon. The data protection duties already apply.
What had to be true
- Safety first: urgent symptoms escalated by rules, never left to a model's judgement.
- Clinician in the loop: the AI drafts, a clinician decides.
- Identifiable data stays home: names, numbers and IDs never reach an offshore model.
- Both languages work, including mixed English and Swahili, tested on every change.
- A living DPIA, updated with each release and backed by evidence.
Architecture
The data flow, end to end
- MessageA patient describes symptoms by app, WhatsApp or SMS
- ScreenRules check for red flags; urgent cases go straight to a clinician
- De-identifyNames, numbers and IDs are replaced with tokens in-country
- DraftThe model asks follow-up questions and drafts a summary and routing
- ReviewA clinician sees the summary, the suggestion and the full history
- RecordThe decision is stored in the in-country record with the model version
The hard part: giving the model context without giving it identities
Good triage needs context: age band, past visits, current medicines, allergies. Shipping the whole record to the model would have been easy and wrong. The team instead exposed clinic systems through small MCP servers running in-country, each offering narrow, read-only tools such as "medicines for this patient token" or "visits in the last 90 days"6. The servers return only the fields the tool needs, keyed by token, so the model can reason about the patient without ever seeing who they are.
De-identification in free text is harder than in records. Patients type names, phone numbers, locations and the names of relatives, often mixing English and Swahili. The gateway combined pattern rules with a small local model for entity detection, tested on real, consented examples in both languages, and failed closed: if it was unsure, the message went to a human rather than to the model. More on securing tool servers in the MCP security checklist.
The question is not "can we send this to the model?" but "what is the least the model needs to be useful?" Build tools that answer exactly that.
How it ran
- Red-flag rules
- Review workflow
- Safety reviews
- Gateway and MCP servers
- Evaluation harness
- Tracing
- DPIA
- Transfer safeguards
- Patient notices
- Red-flag rules
- Draft DPIA
- Clinician workflow
- De-identification gateway
- MCP servers
- Zero-retention model
- AI drafts unseen
- Compare with clinicians
- Fix gaps
- Drafts shown to clinicians
- Evening pilot
- DPIA updated
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Rules before the model for red flags | Let the model spot emergencies | Predictable, testable and explainable |
| De-identify in-country | Send records under a contract | Identifiable health data never leaves |
| Narrow MCP tools | Give the model the whole record | Least data for each question |
| Fail closed on uncertain redaction | Send and hope | A human is cheaper than a breach |
| Shadow mode before assist | Pilot directly with patients | Find errors without risk to care |
What I'd do again, and differently
Again: shadow mode. It showed where the model asked poor follow-up questions in Swahili before any patient saw them. Differently: build the bilingual evaluation set first, not alongside the gateway. It was the slowest part to get right and it drove most design choices. Related: private AI for clinical documentation, private LLMs, evaluation sets, healthcare data architecture and where I work.
If you are bringing AI into healthcare in Kenya or elsewhere in Africa, see my AI architecture work, or analyse your challenge and bring the result to a call.
- 1The Data Protection Act, No. 24 of 2019Kenya Law · 2019
- 2
- 3
- 4Kenya Wants Stronger Privacy Protections for Data Sent to Offshore AI ProvidersTechTrends KE · 6 Aug 2026
- 5African regulators lean on data protection laws as dedicated AI legislation stalls, report findsBusiness and Human Rights Resource Centre · Oct 2026
- 6What is the Model Context Protocol (MCP)?Model Context Protocol · 2026