Replication
Replication, also known as data redundancy, is the process of creating and maintaining identical copies of data across multiple servers or nodes. In cloud platforms, replicas are commonly spread across availability zones to survive data-center-level failures.
Together with sharding, replication is a key technique for scaling databases, and a core technique of distributed databases. Sharding splits data across multiple servers, while replication duplicates data across multiple servers. The two together are what make NoSQL databases horizontally scalable and fault-tolerant, allowing a cluster to grow in capacity and survive node loss without an outage.
Data replication becomes necessary the more that data is distributed across multiple servers, for example in service-oriented or microservices architectures.
In a monolithic architecture, data is typically stored in a single database instance, and joins are typically used to query relational data across multiple tables.
But in a distributed software, those joins turn into network calls. This increases latency, decreasing performance and having implications for the resilience of the system.
Data replication is one possible solution to these problems. Simply, data is replicated to wherever it is needed, so that discrete services are less reliant upon each other. Services therefore work faster by needing to use only local data.
Besides the performance benefits of reducing network calls, data replication also improves the fault tolerance and availability of distributed software, due to reduced dependencies between services. Replicas also spread read load across nodes, so a hot primary is no longer the single bottleneck for queries, raising read throughput without changing the write path.
Data replication is a useful strategy to support the re-architecture of system designs. For example, in the transition from monolithic services to more microservices, data replication can help to reduce the need for scheduled downtime while data migrations are run. A new service can be run against a replicated copy of the monolith’s tables while the cut-over is validated, so the legacy system keeps serving traffic. Once the new service is trusted, it takes ownership of its data and the legacy copy is retired, moving toward one source of truth per service.
Trade-offs are made in terms of data consistency, and the increased system complexity required to manage the data replication and synchronization processes. Storage costs also increase, since every replica is another full copy of the data, and the bandwidth needed to keep copies in step grows with the write rate.
The consistency trade-off is the heart of the CAP theorem. Once data lives on more than one node, a network partition can force a choice between refusing writes to keep copies in step and accepting them at the cost of temporary divergence between replicas. Most replicated stores lean toward the latter, letting copies converge over time rather than blocking writes while a partition heals.
Data synchronization can be achieved using either batch processes or event streams. Batch processes periodically dump and reload snapshots of the source data, which is simple to reason about but leaves replicas stale between runs. Event streams propagate each change as it happens, so replicas track the source with far less lag, at the cost of ordering, delivery, and replay machinery. Some database systems also provide built-in support for data replication across multiple nodes, typically by shipping their write-ahead log or replaying statements row by row. Built-in replication is transparent to applications but usually ties the replicas to the same database engine as the source.
Leaderless and eventually-consistent stores often synchronize replicas with an anti-entropy gossip protocol, which repairs divergence through randomized peer-to-peer exchanges.
Data need not be replicated in the form of its source of truth. A replica can hold a transformed or denormalized projection rather than a verbatim copy. Materialized views are the common case, storing pre-processed data that’s optimized for specific use cases, such as reporting.