Heartbeat
In distributed systems, a heartbeat is a periodic message sent between nodes to signal that the sender is alive and functioning. Each heartbeat carries little payload beyond a timestamp or sequence number, and is repeated at a fixed interval, typically seconds to minutes, depending on how quickly failures must be detected.
The receiver treats the steady arrival of heartbeats as proof of liveness. If heartbeats stop arriving within a timeout window, usually a small multiple of the interval chosen to absorb normal latency and jitter, the receiver declares the sender suspected or failed. The interval and timeout are a single trade-off. Too tight and a slow network produces false positives that mark healthy nodes dead. Too loose and detection lags, delaying recovery.
Without a heartbeat mechanism, it’s hard to quickly detect failures in a distributed system. The consequences are as follows.
- Delayed fault detection and recovery.
- Increased downtime and errors.
- Decreased overall system reliability.
Once a missing heartbeat has flagged a node as failed, the system can take corrective action, most commonly a failover that promotes a redundant peer to take the failed node’s place. In this way heartbeats are the detection half of fault tolerance. They don’t prevent failures, but they make failures visible fast enough for recovery mechanisms to keep the system running.
A heartbeat is a narrower signal than a system health check. A health check gathers metrics about a node’s state, such as CPU, memory, and disk usage, whereas a heartbeat only asserts liveness. The two are complementary. Health checks say how a node is doing; heartbeats say whether it is doing anything at all.
A missing heartbeat is itself a signal. Alerting systems can raise a dead man’s switch alert when heartbeats stop arriving, catching total failures that value-based thresholds would miss.
Heartbeat state can also be propagated through a gossip protocol, so that every node eventually learns which peers have stopped heartbeating and can treat them as failed. This scales failure detection to clusters too large for every node to probe every peer directly.
Leaders elected by consensus algorithms such as Raft send heartbeats to followers to maintain their authority. A follower that stops receiving them starts a new election. Here the heartbeat doubles as both a liveness signal and a lease on leadership.
Caution
A network partition can split a cluster so that each side stops receiving the other’s heartbeats and declares its peers dead. Both sides may then promote standbys and diverge, a split-brain condition that heartbeats alone cannot resolve and that consensus protocols guard against with quorum requirements.