Most engineering organisations know what to do when a service goes down: page someone, restore it, write a postmortem, fix the causes. Far fewer know what to do when an AI feature confidently gives a customer wrong information, leaks something it shouldn't, or treats one group of users worse than another. There is often no alert, no owner and no record, just an unhappy customer and an awkward meeting.
Define what counts
A shared definition is the starting point. The OECD defines an AI incident as an event where the development or use of an AI system results in actual harm, and an AI hazard as one that could plausibly lead to harm. Its harm categories include injury to health, disruption of critical infrastructure, violations of rights or legal obligations, and harm to property, communities or the environment3. The OECD also tracks publicly reported incidents and hazards through its AI Incidents and Hazards Monitor4.
Internally, I use a wider net, because most AI problems never reach the news. A useful internal definition adds near misses: wrong outputs caught by a human, guardrails triggering on real traffic, and quality regressions found in production.
Borrow from SRE, deliberately
Google's SRE book describes blameless postmortems as focusing on contributing causes without indicting individuals, assuming people acted in good faith with the information they had. It warns that a culture of blame stops people bringing problems to light1. The SRE workbook adds that introducing postmortems is as much a cultural change as a technical one2.
That matters even more for AI. If an engineer who reports a model's embarrassing output gets blamed, the next one won't report it, and silent failures are exactly what AI systems produce.
The goal is to make reporting an AI failure feel like a contribution, not a confession. Everything else depends on people speaking up.
What makes AI incidents different
| Ordinary outage | AI incident |
|---|---|
| Usually loud: errors, alerts, downtime | Often silent: the system works but the answer is wrong |
| Reproducible with the same input | Probabilistic; the same prompt may not fail twice |
| Cause is usually a change in code or infrastructure | Cause may be data, prompt, model version, retrieval or user behaviour |
| Fix is a code or config change | Fix often includes evaluation cases, guardrails and product changes |
The playbook
- Make it easy to report. A feedback button in the product, a channel for staff and a clear rule that guardrail triggers on real traffic open a ticket.
- Capture the evidence automatically. Inputs, retrieved context, prompt and model versions, tool calls and outputs, with appropriate privacy controls, as described in LLM observability. Without this, AI postmortems become guesswork.
- Classify severity by harm, using categories like the OECD's, not just by technical impact. A wrong answer to one vulnerable user can outrank an hour of downtime.
- Contain first. Disable the feature, route to a safer model or add a temporary guardrail before investigating. Design these switches in advance.
- Write the postmortem blamelessly, covering what happened, who was affected, contributing causes across data, prompts, models and product, and what made it hard to detect.
- Turn every incident into evaluation cases. The failing inputs join your evaluation set, so the same failure can't quietly return after the next model update.
- Review patterns quarterly. Individual incidents fix bugs; patterns reveal design problems, such as an agent with too many permissions or a retrieval index without access controls.
Regulatory context
Incident handling is increasingly expected by regulators and auditors. Serious incident reporting is part of the EU AI Act for high-risk systems5, and a working process is strong evidence for SOC 2 and for governance programmes in the Gulf. Even where no rule applies, a record of how you learn from failures is one of the most persuasive things you can show an enterprise customer or investor.
How I can help
I set up AI operations for engineering teams, from monitoring and evidence capture to postmortem practice and evaluation loops. As a fractional CTO I can build that culture alongside the systems. Or analyse your challenge and bring the result to a call.
- 1
- 2
- 3
- 4
- 5