- Every kind of change, code, configuration and network, goes through a staged rollout
- Canaries judged by customer-facing signals such as payment success and latency
- Rollback treated as the default response, rehearsed and fast
- Fewer large changes; more small, reversible ones
- Change records that support regulators' expectations on resilience and incident handling
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A Brazilian payment institution offered accounts, cards and instant payments to consumers and small businesses, processing more than a million transactions a day. Customers expected instant payments to complete in seconds at any hour, and partners expected the same. The platform ran on Kubernetes across two cloud regions, with dozens of services owned by several teams.
Its post-incident reviews told a consistent story. The serious incidents rarely came from attacks or hardware. They came from the company's own changes: a configuration value changed everywhere at once, a library upgrade that behaved differently under peak load, a network rule adjusted during maintenance. Code deployments had some safeguards; configuration and network changes had almost none.
Brazil's National Monetary Council requires regulated institutions to maintain a cybersecurity policy with incident response and to manage risks in data processing and cloud services5. The company's resilience depended as much on how it changed its own systems as on any external threat.
Why now. In November 2026, Microsoft attributed a long Microsoft 365 outage to traffic shifts during routine network maintenance exposing a latent bug from an earlier capacity upgrade12. If one of the world's largest operators could be caught by a routine network change, the board asked, what protected a payments platform from the same pattern?
What had to be true
- Every change counted: code, configuration, infrastructure and network changes all go through a controlled path.
- Small first: changes reach a small slice of traffic before everyone.
- Judged by customers: health measured by payment success and latency, not server metrics alone.
- Reversible: rollback is fast, rehearsed and the default when in doubt.
- Visible: anyone can see what changed, where and when, during an incident.
- Still fast: teams ship often; safety comes from smaller changes, not fewer.
Architecture
Code deployments moved to progressive rollouts on Kubernetes: a new version first replaced a small number of pods while the rest kept serving, and only continued if it stayed healthy. Google's SRE guidance describes canarying in exactly these terms: test each change on a small slice of traffic so defects are caught early with limited impact3.

The bigger change was bringing configuration and network changes into the same discipline6. Configuration moved into version control and rolled out by region and by service in stages, with the same health checks as code. Network changes, such as firewall rules, load balancer settings and routing, went through a pipeline that applied them to one zone first and watched payment signals before continuing.
The data flow, end to end
- ProposeA change is recorded with its owner, purpose and rollback plan
- CanaryIt reaches a small slice of traffic or one zone
- JudgePayment success, latency and errors are compared with the baseline
- ExpandIf healthy, it proceeds in waves across services and regions
- HaltIf a signal regresses, rollout stops and rolls back automatically
- LearnHalts and incidents feed back into checks and tests
The hard part: judging health in a payments system
Choosing signals that reflect customers. Server metrics often looked fine while customers suffered. Canary analysis used signals that matched customer experience: payment success rate by payment type and partner, end-to-end latency, error codes returned to apps and depth of downstream queues. The SRE workbook stresses choosing metrics that genuinely indicate problems and comparing the canary against a control rather than a fixed threshold3.
Low-volume paths and latent bugs. A canary on a small slice of traffic can miss problems in rare paths, such as an unusual partner or a refund flow, and latent bugs may only appear under peak load. The team added synthetic transactions for rare paths, kept each wave long enough to cover a busy period for critical changes, and avoided large changes during known peaks such as month-end and holidays.
Rollback as the default. Engineers used to try to fix forward during incidents. The new rule: if a recent change might be involved, roll it back first and investigate afterwards. Rollback was rehearsed regularly, and the change timeline appeared on every dashboard so responders could see what had just changed. Google's release engineering guidance treats reliable, repeatable releases and rollbacks as foundations of reliability4.
Most outages are self-inflicted. Make every change small, watched and reversible, including the configuration and network changes nobody thinks of as releases.
How it ran
- Progressive rollout tooling
- Config and network pipelines
- Automated rollback
- Feature flags
- Health signals per service
- Rollback rehearsals
- Change records
- Zone-by-zone changes
- Peak calendar
- Change policy
- Incident evidence
- Regulatory reporting support
- Change timeline
- Customer signals
- Incident review of causes
- Canary rollouts
- Automatic halt
- Feature flags
- Config as code
- Zone-first network changes
- Same health checks
- Rollback rehearsals
- Peak calendar
- Learning from halts
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Stage configuration and network changes | Stage code only | Many serious incidents start outside code |
| Judge canaries on payment signals | CPU and memory thresholds | Customers feel payments, not servers |
| Compare canary with control | Fixed thresholds | Handles daily traffic patterns |
| Rollback first, investigate later | Fix forward | Shorter incidents, calmer responders |
| Synthetic checks for rare paths | Rely on live canary traffic | Low-volume flows still get tested |
| Peak calendar for large changes | Ship anytime | Reduces risk when impact would be highest |
What I'd do again, and differently
Again: put every change, of every type, on one timeline overlaid on dashboards; it shortened incidents before any automation existed. Differently: bring network engineering in from the start rather than after the code pipeline was done; network changes turned out to carry the most risk and the least tooling. Related: the Microsoft 365 outage playbook, SLOs and error budgets, the payments platform case study and where I work.
If your changes still go out to everyone at once, see my fractional CTO work, or analyse your challenge and bring the result to a call.
- 1Microsoft says network upgrade caused Microsoft 365 outageBetaNews · Nov 2026
- 2
- 3
- 4
- 5Resolução CMN nº 4.893, de 26 de fevereiro de 2021Conselho Monetário Nacional (via ANCORD) · 26 Feb 2021
- 6module_06_rollingupdates2.svg (Kubernetes basics tutorial images)Kubernetes · 2026