Disaster recovery
Disaster recovery (DR) is the set of policies, tools, and procedures for restoring a system’s operations after a disruptive event: a hardware failure, a data center outage, a network partition, data corruption, or a software failure that cascades into an incident. It is one facet of the broader resilience of a distributed system. Where fault tolerance and redundancy aim to keep a system running through a failure, disaster recovery is about limiting how long a failure keeps the system out of service when those measures are not enough.
DR planning is typically framed around two objectives. The recovery time objective (RTO) is the maximum tolerable duration between a failure and the restoration of service. The recovery point objective (RPO) is the maximum tolerable amount of recent data loss, measured backwards from the moment of failure. Together they set the budget for redundancy, failover, and backup strategy. Tighter objectives demand more standby capacity and more frequent replication, at greater cost. RTO and RPO figures are commonly committed contractually in a service level agreement, where breaching them triggers service credits or other remedies.
Traditional disaster recovery methodologies do a good job of mitigating the impact of individual servers, and even whole data centers, going down. They combine redundant hardware and data replication with failover to a standby system, so that a failed component can be routed around or replaced without the system as a whole going down. These techniques address hardware and network failures, which are the classic triggers for a disaster recovery invocation.
Software bugs
DR strategies also need to include deployment strategies, such as blue-green deployments and canary releases, to minimize downtime caused by software failures. Bugs shipped to production, such as a memory leak or an unhandled edge case, are one of the most common causes of incidents and outages in cloud-based software services, and they can be difficult to recover from if not included in disaster recovery planning.
The hazard is that conventional failover offers no protection against them. When the primary system fails, a secondary running an identical copy of the software inherits the same defect, and fails the same way.
A case in point is the outage in August 2023 of the UK’s National Air Traffic Service (NATS). A critical exception occurred when the primary system tried to process a particular flight plan, and was unable to generate a valid route for it. The secondary system kicked in and took over from the failing primary system within 20 seconds. But because the failover system ran an identical copy of the software, the same critical exception occurred, taking down the secondary system too.
The outage was three hours long – the time it took to ship a bug fix – but the knock-on impact lasted three days, disrupted over 700,000 passengers, and cost the industry an estimated £100 million.
A Stand-in is a disaster recovery strategy devised by Monzo that aims to avoid the scenario of failures cascading through all redundant systems due to software errors. Rather than failing over to a replica of the primary, it fails over to a separate, simpler system with different code, data, and infrastructure.
Recovery procedures only help if they work when invoked. Recovery testing exercises them under simulated failure conditions, and practices like chaos engineering extend this testing into live systems. The procedures themselves are typically captured in an emergency runbook.