At around 06:00 UTC on 20 October, a power failure affected part of the Google Cloud infrastructure we use in London. During the outage many Revolut customers couldn’t make payments, withdraw cash, or access the app to check balances.
We understand the stress and inconvenience this will have caused, and apologise for the time it took for services to resume. Even while access to accounts was disrupted, all accounts, customer funds, and data continued to remain safe.
While our systems have been designed to withstand this kind of disruption, on this occasion our backup systems did not take over as expected.
This blog post will provide a technical overview of the root cause, why recovery took longer than it should have, and the actions we're already taking to ensure this doesn't happen again.
What happened?
Revolut operates across several cloud regions. The databases affected by this outage run across three separate data centres (“Zone A/B/C”) in Google Cloud’s London region, with replicas in another region for disaster recovery.
On 20 October at around 06:00 UTC, one data centre (Zone B) lost power and some databases failed to execute zonal failover safely.
We use coordination servers to decide which replica of a database can accept new transactions. These servers operate in groups of five and are spread across three zones, with at least three in each group required to reach quorum.
In some groups, one server was already offline before the power cut, but our monitoring hadn't alerted us. When two more servers in each of those groups went offline in Zone B, only two remained, which isn't enough to reach an agreement. Some tools our engineers relied on to monitor and restore the databases also depended on Zone B, delaying remediation further.
Backup processing allowed some card payments to go through during the outage.
- By 12:40 UTC on 20 October, card authorisations and in-store payments were working again.
- From 15:45 UTC, customers began to regain access to the app, see updated balances, and make transfers. Some payments made during the outage took longer to appear in the app.
- By 17:15 UTC, all remaining services were back online.
What went wrong?
Our database resilience setup involves replicas in three separate London zones. A further replica in another region supports recovery from a wider regional outage. If a zone goes down, the failover manager should have promoted a database replica in another zone. Kubernetes separately reschedules affected applications onto healthy servers.
The failover manager installed in every replica uses a distributed configuration store (DCS) to establish which replica may take over. These coordination servers use a majority vote, called a quorum, to record which database replica is the primary. The servers operate in groups of five and are spread across three zones, with at least three in each group required to reach quorum.
Before the outage, only four of the five DCS servers were available in the affected groups. After Zone B lost power, the two surviving servers could no longer form a quorum. The failover manager couldn't promote a new primary.
We later found that some routine server replacements had failed to restore all five members of a group before the outage. The remaining four could keep working, so customers saw no immediate interruption. However, we weren't alerted that a member was missing while there was still time to restore it.
Early reports suggested that several cloud regions might be affected, including our backup region. Some of our own monitoring tools were offline, so it took us time to establish how far the disruption extended. Once we confirmed that the power failure was confined to London’s Zone B and that Zones A and C were still available, we chose to continue restoring services in those two zones, in line with our recovery plan for a single-zone outage.
Factors that delayed recovery
PostgreSQL recovery
After the DCS quorum was restored, some PostgreSQL clusters remained read-only because replication lag prevented automatic failover. Engineers needed to verify replica state and determine a safe recovery path. We're working directly with Google Cloud to better understand the suspected infrastructure brownout more precisely and how it affected database recovery.
Supporting tool failures
Our primary instance for visualising database metrics was deployed in Zone B. Engineers were forced to diagnose distributed database topologies through logs, command-line utilities, and creation of ad hoc standalone monitoring instances in surviving zones before remediation could fully begin.
What we're changing
To ensure a single availability zone failure can't cause this level of disruption again, we've made some immediate changes and begun work on strategic improvements.
As of 13 November, we've:
- addressed the root cause and resolved underlying degradation in the coordination servers to restore full failover functionality
- conducted a thorough root cause analysis aimed at strengthening our operational resilience and mitigating future risks
- deployed system enhancements to immediately process card reversals following stand-in processing (STIP) backlog resolution
- enhanced our database monitoring infrastructure to ensure availability during single-zone outages
In-flight architectural and operational changes:
- Strengthening zonal failover testing: our disaster recovery testing programme covers a broad range of failure scenarios. We've now extended our zonal failover tests to cover the conditions observed during this incident and increased the frequency of test execution.
- Earlier warning when a redundancy is reduced: we've implemented comprehensive monitoring of DCS cluster membership, voter health and cluster failure tolerance to provide early warning when quorum is at risk. We're also introducing end-to-end monitoring of our alerting pipelines to detect failures in alert generation and delivery. We're adding proactive warnings when we approach our cloud resource quotas and run short of network addresses required for a zonal failover. Our zonal failover exercise will check whether we can bring services back under load.
- More resilient supporting components and toolings: we're improving our supporting tools to make recovery more efficient and reduce the need for manual intervention if a zone goes offline. We're testing these improvements as part of existing recovery exercises.
- Rebalance database and DCS placement across availability zones: we already spread database replicas across zones, but our deployment process favoured one zone when placing the first server in each group. We're changing the placement rule and checking that each group can still operate if any one zone goes offline as part of the zonal failover exercise.
Our commitment
Revolut is built on the expectation that services are available 24/7, without exception. On 20 October, we fell short of that standard.
While the initial trigger was an external infrastructure failure, our architecture should have absorbed it seamlessly. We've addressed the immediate vulnerabilities that prevented failover, and we're holding our disaster recovery pipelines to a higher standard. We appreciate our customers' continued trust in us as we strengthen our operational resilience to ensure this doesn't happen again.
