Case study · Reliability and release engineering

Change without fear: progressive rollouts and fast rollback for a Brazilian payments platform

A Brazilian payment institution processed more than a million transactions a day, most of them instant payments that customers expect to complete in seconds. Most of its incidents were not caused by attackers or hardware, but by its own changes: a configuration tweak, a library upgrade, a network adjustment. After a major provider's long outage was traced to a routine network change exposing a latent bug, the company decided to make every change small, observable and reversible.

Case studyPayments · BrazilSumanta Mukhopadhyay9 min read
#platform#kubernetes#observability
CASE STUDY · CASE STUDY · RELIABILITY AND RELEASE ENGINEERING16 Nov 2026 · 15:50 GST
At a glance
SectorPayments · Brazil
My roleFractional CTO and platform architect
EngagementAssessment, architecture and delivery leadership
ScopeChange inventory, progressive delivery, canary analysis, feature flags, configuration and network change control, rollback, incident learning
What changed
  • Every kind of change, code, configuration and network, goes through a staged rollout
  • Canaries judged by customer-facing signals such as payment success and latency
  • Rollback treated as the default response, rehearsed and fast
  • Fewer large changes; more small, reversible ones
  • Change records that support regulators' expectations on resilience and incident handling
Kubernetes with progressive rolloutsCanary analysis on service level indicatorsFeature flagsConfiguration as code with staged rolloutNetwork change pipelineOpenTelemetry metrics and tracesAutomated rollbackChange calendar and freeze rules

Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.

The situation

A Brazilian payment institution offered accounts, cards and instant payments to consumers and small businesses, processing more than a million transactions a day. Customers expected instant payments to complete in seconds at any hour, and partners expected the same. The platform ran on Kubernetes across two cloud regions, with dozens of services owned by several teams.

Its post-incident reviews told a consistent story. The serious incidents rarely came from attacks or hardware. They came from the company's own changes: a configuration value changed everywhere at once, a library upgrade that behaved differently under peak load, a network rule adjusted during maintenance. Code deployments had some safeguards; configuration and network changes had almost none.

Brazil's National Monetary Council requires regulated institutions to maintain a cybersecurity policy with incident response and to manage risks in data processing and cloud services5. The company's resilience depended as much on how it changed its own systems as on any external threat.

Why now. In November 2026, Microsoft attributed a long Microsoft 365 outage to traffic shifts during routine network maintenance exposing a latent bug from an earlier capacity upgrade12. If one of the world's largest operators could be caught by a routine network change, the board asked, what protected a payments platform from the same pattern?

What had to be true

  • Every change counted: code, configuration, infrastructure and network changes all go through a controlled path.
  • Small first: changes reach a small slice of traffic before everyone.
  • Judged by customers: health measured by payment success and latency, not server metrics alone.
  • Reversible: rollback is fast, rehearsed and the default when in doubt.
  • Visible: anyone can see what changed, where and when, during an incident.
  • Still fast: teams ship often; safety comes from smaller changes, not fewer.

Architecture

Changes
CodeConfigurationInfrastructureNetworkpreviously ungoverned
Pipeline
Change recordwho, what, whyStaged rolloutcanary, then wavesFeature flagsdecouple deploy from release
Judgement
Canary analysispayment success, latency, errorsAutomatic halton regression
Recovery
Automated rollbackKill switchesper feature
Evidence
Change timelineoverlaid on dashboardsIncident reviews
All change types share one path: record, stage, judge, and roll back if needed.

Code deployments moved to progressive rollouts on Kubernetes: a new version first replaced a small number of pods while the rest kept serving, and only continued if it stayed healthy. Google's SRE guidance describes canarying in exactly these terms: test each change on a small slice of traffic so defects are caught early with limited impact3.

Kubernetes rolling update diagram: one pod has been replaced by the updated app with a new IP address while the other pods behind the same Service still run the previous version
A rolling update replaces one pod at a time while the Service keeps routing to healthy pods: the starting point for safe change. Source: Kubernetes documentation · CC BY 4.0

The bigger change was bringing configuration and network changes into the same discipline6. Configuration moved into version control and rolled out by region and by service in stages, with the same health checks as code. Network changes, such as firewall rules, load balancer settings and routing, went through a pipeline that applied them to one zone first and watched payment signals before continuing.

The data flow, end to end

  1. ProposeA change is recorded with its owner, purpose and rollback plan
  2. CanaryIt reaches a small slice of traffic or one zone
  3. JudgePayment success, latency and errors are compared with the baseline
  4. ExpandIf healthy, it proceeds in waves across services and regions
  5. HaltIf a signal regresses, rollout stops and rolls back automatically
  6. LearnHalts and incidents feed back into checks and tests
The same flow for code, configuration and network changes.

The hard part: judging health in a payments system

Choosing signals that reflect customers. Server metrics often looked fine while customers suffered. Canary analysis used signals that matched customer experience: payment success rate by payment type and partner, end-to-end latency, error codes returned to apps and depth of downstream queues. The SRE workbook stresses choosing metrics that genuinely indicate problems and comparing the canary against a control rather than a fixed threshold3.

Low-volume paths and latent bugs. A canary on a small slice of traffic can miss problems in rare paths, such as an unusual partner or a refund flow, and latent bugs may only appear under peak load. The team added synthetic transactions for rare paths, kept each wave long enough to cover a busy period for critical changes, and avoided large changes during known peaks such as month-end and holidays.

Rollback as the default. Engineers used to try to fix forward during incidents. The new rule: if a recent change might be involved, roll it back first and investigate afterwards. Rollback was rehearsed regularly, and the change timeline appeared on every dashboard so responders could see what had just changed. Google's release engineering guidance treats reliable, repeatable releases and rollbacks as foundations of reliability4.

Most outages are self-inflicted. Make every change small, watched and reversible, including the configuration and network changes nobody thinks of as releases.

How it ran

Platform
  1. Progressive rollout tooling
  2. Config and network pipelines
  3. Automated rollback
Service teams
  1. Feature flags
  2. Health signals per service
  3. Rollback rehearsals
Network and operations
  1. Change records
  2. Zone-by-zone changes
  3. Peak calendar
Risk and compliance
  1. Change policy
  2. Incident evidence
  3. Regulatory reporting support
Network operations adopting staged changes was the biggest cultural shift.
Phase 1See
  • Change timeline
  • Customer signals
  • Incident review of causes
Phase 2Stage code
  • Canary rollouts
  • Automatic halt
  • Feature flags
Phase 3Stage config and network
  • Config as code
  • Zone-first network changes
  • Same health checks
Phase 4Habit
  • Rollback rehearsals
  • Peak calendar
  • Learning from halts
Seeing every change on one timeline came first; it made the case for everything else.

Key decisions and trade-offs

DecisionAlternativeWhy
Stage configuration and network changesStage code onlyMany serious incidents start outside code
Judge canaries on payment signalsCPU and memory thresholdsCustomers feel payments, not servers
Compare canary with controlFixed thresholdsHandles daily traffic patterns
Rollback first, investigate laterFix forwardShorter incidents, calmer responders
Synthetic checks for rare pathsRely on live canary trafficLow-volume flows still get tested
Peak calendar for large changesShip anytimeReduces risk when impact would be highest

What I'd do again, and differently

Again: put every change, of every type, on one timeline overlaid on dashboards; it shortened incidents before any automation existed. Differently: bring network engineering in from the start rather than after the code pipeline was done; network changes turned out to carry the most risk and the least tooling. Related: the Microsoft 365 outage playbook, SLOs and error budgets, the payments platform case study and where I work.

If your changes still go out to everyone at once, see my fractional CTO work, or analyse your challenge and bring the result to a call.

References
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
    Resolução CMN nº 4.893, de 26 de fevereiro de 2021Conselho Monetário Nacional (via ANCORD) · 26 Feb 2021
  6. 6
FAQ

Common questions

What is a canary release?

Rolling a change out to a small share of traffic or servers first, comparing its behaviour against the existing version on key signals, and only continuing if it is healthy. Google's SRE guidance describes canarying as a way to catch defects early while limiting impact.

Why include configuration and network changes in progressive rollout?

Because many serious outages come from configuration and network changes rather than code. Microsoft's November 2026 Microsoft 365 outage, for example, was attributed to network maintenance exposing a latent bug from an earlier upgrade.

What signals should decide whether a payments canary is healthy?

Customer-facing signals: payment success rate, end-to-end latency, error rates by payment type and partner, and downstream queue depth, rather than only server CPU or memory.

Keep reading

Related

Open to advisory, fractional CTO & select builds

Let's talk about what you're building.

The fastest route is a free 30-minute call. Prefer to write? Send a message and I'll reply personally within two business days.

Start a conversation

How would you like to reach me?

Book a 30-min callFastest. Pick any open slot on my calendar.→ Write in GmailOpens a pre-filled draft in your browser.→ Use my email appOutlook, Apple Mail or your default client.→

I reply personally within two business days. Time zones from the Americas and Europe to the Middle East and Asia-Pacific.