Skip to content
All case studies

Case study 052024

Multi-region Failover

Warm-standby disaster recovery for a payments platform, with a one-command failover rehearsed every quarter.

Role
Lead engineer
Team
Platform team of 4 + service owners
Timeline
6 months
Focus
Platform, Backend

4 h → 9 min

recovery time

< 1 s

data-loss window

4

production failover drills passed

Traffic runs in the primary region while data replicates to a warm standby; one rehearsed command promotes the standby and shifts traffic.Customers to Traffic manager; Traffic manager to Primary region (live); Traffic manager to Standby region (on failover); Primary region to Postgres primary; Standby region to Postgres standby; Postgres primary to Postgres standby (replication < 1 s); Failover controller to Traffic manager (shift)Customersweb · mobile · APITraffic managerhealth-checked routingPrimary regionactiveStandby regionwarm · scaled downPostgres primarywritesPostgres standbypromotableFailover controllerone audited commandliveon failoverreplication < 1 sshift
Traffic runs in the primary region while data replicates to a warm standby; one rehearsed command promotes the standby and shifts traffic.

The problem

A regional outage would have stopped every payment for an unknown length of time. The recovery document was two years old, recovery-time estimates were guesses, and banking partners were asking for tested recovery objectives.

What I owned

  • Recovery objectives agreed with the business: under a second of data loss and under 15 minutes of downtime for money movement.
  • Terraform for the standby region, plus replication for Postgres, Kafka and object storage.
  • The failover controller: drain, promote, shift traffic and verify, as one audited command.
  • Quarterly game days, the incident-commander rota and the runbooks behind them.

Key decisions

  1. 1

    Warm standby, not active-active

    Active-active would have meant rewriting the ledger for multi-writer consistency. A warm standby met the recovery targets at a fraction of the risk and cost.

  2. 2

    Tier services by what an outage costs

    Money movement got the strict targets. Reporting and back-office tools recover later, which kept the standby region small and cheap.

  3. 3

    A drill isn't done until it's boring

    Every quarter we fail over production traffic on purpose. Each drill produces fixes, and by the third one there were no manual steps left.

Outcome

Recovery is now a rehearsed procedure with a measured time, not a document. The tested numbers closed two partner security reviews, and the standby costs less than 20% of the primary region.

  • AWS
  • Terraform
  • PostgreSQL
  • Kafka
  • Kubernetes
  • Go