Resilience

Resilience is the ability of a system to continue functioning, in a degraded or limited capacity, in the presence of failure – rather than failing completely. It is closely related to, but distinct from, fault tolerance: fault tolerance is about preventing a failure from causing harm at all, while resilience accepts that failures will happen and focuses on the system’s ability to absorb them and keep operating, or to recover quickly once they occur.

Resilience is a particularly important quality attribute of distributed software, where the number of independent components – and therefore the number of independent ways things can fail – is much higher than in a single-node system. Several design choices contribute to resilience:

• Isolation between components, so that a failure in one does not cascade into others.
• Circuit breakers and bulkheads, which contain the blast radius of a failing dependency.
• Retries with backoff, for transient failures that are likely to resolve on their own.
• Redundancy, so that a failed component can be replaced or routed around without the system as a whole going down.
• Disaster recovery planning, for the failure modes that redundancy alone cannot absorb.

Architectures with a high degree of decoupling between components – such as peer-to-peer networks, which have no single point of failure, and the saga pattern, which avoids the blocking failure modes of two-phase commit – tend to exhibit greater resilience than tightly-coupled alternatives, at the cost of additional design and operational complexity.