- Every critical business service mapped to the SaaS it depends on, including identity
- Recovery targets per service, with workarounds that fit within them
- A way to coordinate incidents and reach customers when email and chat are down
- Break-glass access that works without the main identity provider
- Early warning from independent checks rather than staff reports
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A Singapore digital insurer sold travel, health and home policies online. Its policy administration, claims and payments ran on its own cloud estate across two availability zones, with tested failover. Everything else ran on one productivity suite: email, chat, video calls, shared documents, the identity provider staff used to sign in to every other system, and the AI assistant built into it. Customer notices, incident bridges and even the runbooks for failing over the core platform lived there.
MAS's business continuity management guidelines expect financial institutions to identify their critical business services and set, for each, a service recovery time objective: how long it can be disrupted before severe impact, with the recovery times of underlying functions and dependencies fitting inside it23. The insurer had done this for its platforms. It had not traced the dependencies through the productivity suite.
Why now. On 12 November 2026, a long Microsoft 365 outage disrupted Outlook, Exchange Online and Teams for much of a working day; Microsoft attributed it to servers failing health checks and dropping out of service, with no evidence of an attack1. Staff across the industry found they could not email customers, join incident calls or open runbooks. The insurer's board asked a simple question: what would we have done?
What had to be true
- Dependencies known: each critical service mapped to the SaaS it relies on, identity included.
- Targets met: workarounds that keep each service within its recovery target.
- Coordination possible: a way to run an incident without email, chat or shared documents.
- Customers reachable: notices that do not depend on the failed suite.
- Access preserved: administrators can reach critical systems if the identity provider is down.
- Early warning: the insurer knows about an outage before staff start complaining.
Architecture
The first deliverable was a dependency map for each critical business service, built with the people who run them. It showed that the identity provider was the biggest single point of failure: if staff could not sign in, they could not reach the claims system, the cloud console or the paging tool, even though those were healthy.
The data flow, end to end
- DetectSynthetic checks from several regions see the suite failing
- PageOn-call staff are paged through a separate paging service
- CoordinateThe incident moves to the out-of-band messaging tool
- DecideLeaders choose workarounds per service, using offline runbooks
- CommunicateCustomers and partners are told via the status page and SMS
- RecoverNormal tools return; the incident record feeds the next exercise
Detection used the OpenTelemetry Collector's HTTP check receiver, run from small collectors in more than one region, to probe the suite's sign-in, mail and chat endpoints alongside the insurer's own services4. Results went to the observability platform the insurer already used, which did not depend on the suite.
The hard part: identity and the fallback nobody uses
Break-glass without a back door. Emergency accounts that bypass the main identity provider are exactly what attackers want. The insurer kept a small number, each tied to a named role, protected with hardware security keys, stored with sealed procedures, monitored for any use and tested on a schedule. Any use outside an exercise raised an immediate alert to security.
Keeping fallbacks alive. A fallback channel that nobody opens for months will not work when needed: accounts expire, numbers change, people forget. The insurer used the out-of-band messaging tool for real on-call handovers every week, synchronised its contact lists from the HR system daily, and printed a short card of first steps for on-call staff.
Regulatory duties without email. Incident reporting to MAS and notices to customers normally went out by email. The runbook named an alternative route for each, and the compliance team rehearsed it, so a suite outage would not also become a reporting failure.
Resilience plans usually cover your systems failing. The outage that hurts is often the supplier your plan quietly depends on: the one hosting the plan itself.
How it ran
- Dependency maps
- Synthetic checks and paging
- Break-glass access
- Workarounds per service
- Offline runbooks
- Weekly use of fallbacks
- Recovery targets
- Regulatory reporting routes
- Exercise evidence
- Status page
- SMS templates
- Partner contacts
- Critical services
- SaaS dependencies
- Identity paths
- Out-of-band channel
- Break-glass access
- Synthetic checks
- Offline runbooks
- Customer templates
- Reporting routes
- Tabletop
- Live switch to fallbacks
- Lessons into plans
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Fallbacks on different providers | Backup features of the same suite | One outage cannot take out both |
| Use fallbacks weekly | Keep them for emergencies only | Unused tools fail when needed |
| Small, monitored break-glass set | No emergency access, or many accounts | Access when needed without a standing back door |
| Independent synthetic checks | Rely on the provider's status page | Earlier warning and your own evidence |
| Offline runbooks and printed first steps | Runbooks only in shared documents | Available when the suite is not |
| Workarounds sized to recovery targets | Generic "wait for the provider" | Meets MAS expectations per service |
What I'd do again, and differently
Again: map dependencies service by service with the people who run them; it surfaced the identity dependency nobody had written down. Differently: include the AI assistant in the map from the start; some teams had begun relying on it for routine customer replies, and its outage was a gap in its own right. Related: the 13 November tech news, the AI provider resilience case study, SLOs and error budgets and the Singapore page.
If a single SaaS outage would stop you coordinating an incident or reaching customers, see my fractional CTO work, or analyse your challenge and bring the result to a call.
- 1Microsoft 365 Outage Disrupts Outlook, Exchange Online and TeamsSecurityWeek · 12 Nov 2026
- 2Business Continuity Management GuidelinesMonetary Authority of Singapore · Jun 2022
- 3MAS issues revised Business Continuity Management Guidelines for FIsAllen & Gledhill · 2022
- 4Portable Synthetic HTTP Testing with OpenTelemetryOpenTelemetry blog · 5 Oct 2023