TS-6: Distributed system design

This technical standard covers the design of software systems composed of multiple independently deployable components that communicate over a network: consistency and availability trade-offs, consensus and coordination, delivery semantics and idempotency, distributed transactions, service topology, and system-level resilience. It also covers the concerns that only appear at the scale of many services — sizing, migration execution, and the ongoing evaluation of a system’s technology choices.

This standard complements, rather than repeats, TS-5: Application architecture, which covers the architecture of a single application and the point at which a service is extracted from one; and TS-46: Distributed data and caching, which covers the mechanics of keeping distributed data consistent. See also TS-20: Network APIs and TS-23: Messages and events for the design of individual network calls and messages, and TS-49: Cloud platform engineering for the surrounding platform infrastructure.

Fundamentals

A distributed system is a set of independently deployable components — services, processes, or machines — that communicate over a network to behave as one coherent system. Almost every non-trivial application is distributed in this sense: a web application talking to a database is already a distributed system of two components. This standard covers the concerns that emerge once a system crosses process and network boundaries, wherever those boundaries fall.

This is the connective standard for distributed-systems concerns that cut across other standards: TS-5: Application architecture covers the shape of a single application and the point at which services are extracted from it; TS-46: Distributed data and caching covers how data specifically is kept consistent once copied or replicated; TS-20: Network APIs and TS-23: Messages and events cover the mechanics of individual calls and messages. This standard covers the system-wide properties and failure modes that emerge only when many such components operate together: consistency and availability trade-offs, consensus, delivery semantics, distributed transactions, service topology, and the organizational mechanics of running and evolving a fleet of services.

The central problem: partial failure

A single process fails completely or not at all. A distributed system fails partially — one node, one link, or one dependency can fail while the rest of the system keeps running, and the components that did not fail cannot always tell the difference between "the other side crashed," "the other side is slow," and "the message was lost in transit." This ambiguity is the single hardest problem in distributed systems design, and nearly every pattern described in this standard — timeouts, retries, idempotency, consensus, sagas — exists to manage it.

Code that has only ever run on one machine tends to assume calls either succeed or throw immediately. Code written for a distributed system MUST treat every call to another component as a call that can fail slowly, fail silently, fail partially, or succeed after the caller has already given up and moved on. Designing for this from the outset is cheaper than retrofitting it after an incident.

The fallacies of distributed computing

L. Peter Deutsch and colleagues at Sun Microsystems catalogued eight false assumptions that engineers new to distributed systems invariably make. Each one is a fallacy precisely because the system will work as assumed in development and under light load, then fail once it meets production conditions — so the fallacies are dangerous by default, not by exception.

  • The network is reliable. Packets are dropped, connections reset, and requests silently disappear. Every network call MUST have a defined behavior for the case where no response arrives at all.
  • Latency is zero. A network round trip is orders of magnitude slower than an in-process call, even on a fast local network. Code that treats a remote call like a local one — calling it in a loop, or synchronously on a hot path — will not scale.
  • Bandwidth is infinite. Payload size, serialization cost, and the number of calls all have a real, billable, and rate-limited cost.
  • The network is secure. An internal network is not a trust boundary. Traffic between services is interceptable and MUST be authenticated and, where it carries sensitive data, encrypted.
  • Topology doesn’t change. Instances are added, removed, and rescheduled continuously in any modern deployment platform. Code MUST NOT hard-code addresses or assume a fixed set of peers.
  • There is one administrator. Large systems cross team, and sometimes organizational, boundaries. Coordinating a breaking change requires negotiation, not a unilateral decision.
  • Transport cost is zero. Serialization, connection setup, load balancing, and encryption all consume real CPU and money at scale, and MUST be accounted for in capacity planning.
  • The network is homogeneous. Real networks mix protocols, MTUs, and latency characteristics. Code SHOULD NOT assume the same performance characteristics on every hop.

Treat this list as a design checklist. A component that has not been reviewed against every fallacy on this list has not been reviewed for production readiness in a distributed system.

Reasoning about systems that defy understanding

Below a certain scale, a distributed system’s behavior can be understood deductively: read the code, trace the call graph, reason about what will happen. Past that scale — many independently-deployed services, evolving independently, under concurrent load — the state space becomes too large to hold in one engineer’s head, and exhaustive deductive reasoning about why a given failure occurred stops being a practical strategy.

The practical response is to switch strategy rather than push harder on the same one. When a system is too complex to reason about deductively:

  1. Invest in empiricism over deduction. Treat the running system as the source of truth. Observe its actual behavior — through logs, metrics, and traces — rather than trying to predict it from first principles. See TS-57: Logging, monitoring, observability.
  2. Treat behavior statistically, not individually. At scale, some requests will always fail for idiosyncratic reasons. Chasing the root cause of every individual failure does not scale; tracking failure rates and their trend does.
  3. Prioritize system-level resilience over root-causing every component failure. A system designed to tolerate the failure of any single component is more valuable than a system in which every past failure has been individually diagnosed and patched, because it also survives the next failure mode, which has not been seen yet. The resilience patterns in this standard — timeouts, retries, circuit breakers, bulkheads, graceful degradation — exist to make this the default outcome rather than the exception.

This does not excuse skipping root-cause analysis where it is tractable — it is a statement about where to invest limited engineering effort once a system has grown past the point where every failure can be individually understood.

Managing state

State is the hardest part of distributed systems design. A component with no state of its own can be killed, restarted, rescheduled, or duplicated at will without any coordination — the platform can throw it away and stand up a replacement without asking anyone’s permission. A component that owns state cannot: every one of those same operations now has to reckon with what happens to the data, and most of the hard problems covered elsewhere in this standard — Consistency and availability, Consensus and coordination, Distributed transactions and sagas — exist only because state has to live somewhere and be kept correct while the system around it keeps changing shape.

It follows that the best service, all else being equal, is a stateless one. "Stateless" here means the service holds no data of record between requests — no data that would be lost, or would leave the system inconsistent, if the instance handling a request were killed the moment it finished responding. A stateless service can be scaled horizontally by adding instances, failed over by killing an unhealthy instance and starting another, and deployed by replacing every instance at once, all without a coordination protocol, because no instance holds anything the others don’t already have equal claim to.

Minimize the number of stateful components

The corollary is a system-design principle, not just a per-service one: minimize the number of components in the system that are stateful. In practice, this means a small number of services own a database — or another system of record — and every other service in the system is purely stateless, deriving everything it needs from a request, a message, or a call to one of those few stateful services.

This is not a call to build a single monolithic data store. It is a call to be deliberate about where state is allowed to accumulate, and to resist it accumulating anywhere else by default:

  • Cache is not an exemption. A service that holds a local, in-memory cache is not stateless unless the system’s correctness does not depend on that cache surviving — see TS-46: Distributed data and caching for the mechanics of keeping any state, cached or otherwise, consistent once it exists in more than one place.
  • Session and workflow state count. In-memory session data, sticky-session routing, and long-running in-process workflow state are all state, even though they are easy to overlook because they don’t sit in a database. A service that requires the same instance to handle every request in a sequence is no longer freely interchangeable, which defeats the point.
  • Pushing state to the edges is not free of design decisions. State that is pushed to the client, to a message broker, or to a small number of dedicated stateful services still has to be designed, secured, and made consistent. Minimizing the count of stateful components concentrates that design effort onto a small, well-understood set of components, rather than eliminating the effort altogether.

New services SHOULD be designed stateless by default. Introducing a new system of record — a new service that owns its own database — is a deliberate architectural decision, not an incidental side effect of a service needing somewhere to put data, and SHOULD be reviewed with the same scrutiny as any other decision that expands the system’s stateful surface area (see TS-5: Application architecture for the application-level considerations around introducing a new data store).

This principle also shapes topology: see Service topology: discovery, gateways, and service mesh for how a service’s stateless-or-stateful status affects how it is deployed, scaled, and discovered.

Consistency and availability

Once data or state exists in more than one place — which is the normal condition in a distributed system — a system MUST make an explicit, documented choice about what happens when the copies cannot communicate. This section covers the theory behind that choice. For the mechanics of implementing a specific consistency guarantee (write-ahead logging, locking, versioning), see TS-46: Distributed data and caching.

The CAP theorem

Eric Brewer’s CAP theorem states that a distributed data store can provide at most two of the following three guarantees simultaneously:

  • Consistency. Every read receives the most recent write, or an error — never stale data.
  • Availability. Every request receives a non-error response, though not necessarily containing the most recent write.
  • Partition tolerance. The system continues to operate despite an arbitrary number of messages being dropped or delayed between nodes.

Network partitions are a fact of distributed systems, not an edge case to be designed around, so partition tolerance is not optional for a system with more than one node — the real choice CAP presents is between consistency and availability during a partition:

  • CP (consistent, partition-tolerant). The system refuses to serve requests it cannot guarantee are consistent, sacrificing availability during a partition. Appropriate where serving stale or conflicting data is worse than serving an error — financial ledgers, inventory counts, distributed locks.
  • AP (available, partition-tolerant). The system keeps serving requests during a partition, accepting that some responses may be stale, sacrificing strong consistency. Appropriate where availability matters more than strict correctness — product catalogs, social feeds, most read-heavy consumer-facing systems.

This choice MUST be made explicitly and per data set, not implicitly by whichever database happened to be selected. A single system commonly needs both: CP for a payments ledger, AP for the product catalog displayed alongside it.

PACELC: the trade-off that exists even without a partition

CAP only describes behavior during a partition, which is a relatively rare event. PACELC extends it with the trade-off that exists at all other times: if there is a *P*artition, choose between *A*vailability and *C*onsistency (as in CAP); *E*lse, in normal operation, choose between *L*atency and *C*onsistency.

Even when the network is healthy, a system that wants strong consistency across replicas MUST coordinate between them before acknowledging a write, which adds latency proportional to the number of replicas and their physical distance from each other. A system that wants the lowest possible latency MUST relax that coordination and accept that replicas may briefly diverge. PACELC is the more complete framework for evaluating a distributed data store or consensus system, because the latency/consistency trade-off applies during the overwhelming majority of a system’s uptime, not just during partitions.

Consistency models

"Consistency" is not binary. In descending order of strength:

  1. Strict/linearizable consistency. Every operation appears to take effect instantaneously at some point between its invocation and its response, and all nodes agree on that single global order. The strongest, most expensive guarantee; typically reserved for a small amount of coordination state (leader election, distributed locks), not for application data at scale.
  2. Sequential consistency. All nodes see operations in the same order, but that order is not required to match real-time invocation order.
  3. Causal consistency. Operations that are causally related (a write that happens after a read that observed a prior write) are seen in that order by every node; unrelated operations may be seen in different orders on different nodes. Strong enough to prevent a user from seeing a reply before the comment it replies to, at a fraction of the cost of linearizability.
  4. Eventual consistency. Given no new writes, all replicas will eventually converge on the same value, with no bound on how long convergence takes and no guarantee about the order in which intermediate states are observed. The weakest and cheapest guarantee, and the default for AP systems.

A system SHOULD choose the weakest consistency model that its use case can tolerate, and no weaker. Defaulting every data set to strong consistency pays an availability and latency cost that most data does not need; defaulting every data set to eventual consistency pushes reasoning about staleness into every consumer of that data, which is rarely what a system designer actually intended.

Important

Consistency model and delivery guarantee are different axes. A message broker can guarantee at-least-once delivery (see [Idempotency and delivery semantics]) while the state built from those messages is only eventually consistent, or vice versa. Evaluate the two independently.

Consensus and coordination

Some decisions in a distributed system cannot be made by a single node acting alone: which replica is the leader, which node holds a lock, whether a transaction committed. These decisions require consensus — getting a set of nodes, any of which may fail, to agree on a single value.

When consensus is actually needed

Consensus is expensive: every decision requires a round trip to a majority of participants, so it adds latency and reduces availability during a partition (a minority partition cannot make progress at all — see Consistency and availability). It SHOULD be reserved for the small amount of state where getting it wrong is unacceptable — leader election, distributed locks, configuration that must be seen identically by every node, commit decisions for a transaction — not used as the default coordination mechanism for application data. Most application state SHOULD be owned by exactly one service and one datastore, avoiding the need for consensus about it entirely.

Quorum systems

A quorum system tolerates the failure of a minority of nodes by requiring every read and write to touch a majority. For a cluster of 2f + 1 nodes, up to f nodes can fail (or be unreachable) while the system continues to make progress, because any two majorities are guaranteed to overlap in at least one node. This is why consensus clusters are near-universally sized as odd numbers (3, 5, 7) — an even-sized cluster adds a node without adding fault tolerance, because the majority threshold rises with it.

A cluster that loses its majority (a "minority partition") MUST stop accepting writes rather than accept them and risk an unrecoverable conflict once the partition heals. This is the availability cost of a CP system, applied concretely.

Leader election and replicated logs

Raft and Paxos are the two consensus algorithms most systems build on; Raft is more widely implemented in modern infrastructure (etcd, Consul, CockroachDB, Kafka’s KRaft mode) specifically because it was designed to be easier to reason about and implement correctly than Paxos.

Raft decomposes consensus into two separable problems:

  1. Leader election. Nodes hold randomized-timeout elections; a candidate that wins votes from a majority becomes leader for a fixed term. Only one leader can exist per term, because winning requires a majority and any two majorities overlap.
  2. Log replication. All writes go through the leader, which appends them to a replicated log and only acknowledges a write once a majority of followers have persisted it. A log entry acknowledged by a majority is committed and will survive the failure of any minority of nodes, including the leader.

A team building a new distributed system SHOULD NOT implement Raft or Paxos from scratch. Use a proven, off-the-shelf consensus system — etcd, ZooKeeper, Consul — or a database that has consensus built in (CockroachDB, Cloud Spanner, several managed Kafka offerings), rather than re-deriving one of the hardest-to-get-right areas of distributed systems.

Distributed locks

A distributed lock is the most common consumer-facing use of a consensus system: it lets otherwise-independent processes coordinate mutually exclusive access to a resource. A distributed lock MUST have a lease with an expiry — never an indefinite hold — so that a process that crashes while holding the lock does not block every other process forever.

A lease-based lock does not, by itself, guarantee mutual exclusion under all failure modes: a process can be paused (garbage collection, virtual machine migration, CPU starvation) for longer than the lease timeout, have its lease expire and be granted to another process, then resume and act as though it still holds the lock. Where the action guarded by the lock is not itself idempotent or safe to perform twice, the lock MUST be paired with a fencing token — a monotonically increasing number issued with each lease grant, which the protected resource checks and rejects if it has already seen a higher token. The lock alone is a performance optimization to reduce contention; the fencing token is what actually guarantees correctness.

Coordination without consensus

Not every coordination problem needs a consensus system. Weaker, cheaper techniques are often sufficient:

  • Optimistic concurrency (versioning). Readers and writers proceed without locking, and a write is accepted only if the version it read still matches; see TS-46: Distributed data and caching for the mechanics. Avoids the cost of consensus entirely where conflicts are rare.
  • Idempotent operations. If an operation is safe to perform more than once, coordination to prevent duplicate execution is unnecessary — see Idempotency and delivery semantics.
  • Single-writer ownership. If exactly one service or one partition owns a piece of state, there is nothing to coordinate — the owner is the sole authority by construction. This is the default a system SHOULD aim for before reaching for a coordination mechanism.

Reach for consensus only after these cheaper options have been ruled out.

Idempotency and delivery semantics

A caller that does not receive a response cannot distinguish between "the request was never received," "the request was received but the response was lost," and "the request is still being processed" (see Fundamentals). This ambiguity is unavoidable, so every distributed system MUST decide, deliberately, what it does when a caller retries under this uncertainty.

The three delivery semantics

  • At-most-once. A request is sent once and never retried. Simple, but any failure is a permanent failure — the safest default only where a missed operation is genuinely tolerable (best-effort telemetry, non-critical notifications).
  • At-least-once. A request is retried until it is acknowledged, which means it may be delivered, and acted on, more than once. This is the semantic every distributed system gets by default the moment it adds retries, and it is the semantic assumed throughout TS-23: Messages and events.
  • Exactly-once. The request is guaranteed to be acted on exactly one time, no matter how many times it is retried. True exactly-once delivery — at the network layer — is not achievable in an asynchronous distributed system: the caller can never be certain the callee has not already acted, so it can never be correct to simply stop retrying. What systems that advertise "exactly-once" actually provide is at-least-once delivery combined with idempotent processing at the receiver, which produces an exactly-once effect even though delivery itself happened more than once.

A system SHOULD default to at-least-once delivery plus idempotent processing. It is the only one of the three that is both achievable and safe by default.

Idempotency

An operation is idempotent if performing it more than once produces the same result, and the same externally visible effect, as performing it once. Every operation that can be retried — which, per the above, is every operation in a distributed system — MUST be made idempotent, or MUST be wrapped in a mechanism that makes retrying it safe.

The standard mechanism is an idempotency key: the caller generates a unique identifier (a UUID is RECOMMENDED) for each logical operation, distinct from any identifier generated by the retry itself, and sends it with every attempt of that operation. The receiver:

  1. Checks whether it has already processed that key.
  2. If yes, returns the stored result of the original attempt without re-executing the operation.
  3. If no, executes the operation, stores the result keyed by the idempotency key, and returns it.

This turns "retry until acknowledged" into a safe default regardless of how many times the request is retried, or how many of those retries actually reach the receiver. Idempotency keys SHOULD be retained for a bounded window (24 hours is a common default) — long enough to cover realistic retry scenarios, not indefinitely, since indefinite retention turns the key store into an ever-growing liability.

Where an operation is naturally idempotent — a PUT that sets a resource to an absolute value, a database upsert keyed on a natural key — no separate idempotency key mechanism is needed. Reserve the explicit key mechanism for operations that are not naturally idempotent by construction, most commonly anything that creates a new record or has an external side effect (charging a payment, sending an email, incrementing a counter).

Ordering

At-least-once delivery does not imply ordered delivery: retries, multiple delivery paths, and concurrent producers can all cause a receiver to observe operations out of the order they were issued. A component MUST NOT assume that messages, events, or requests arrive in the order they were sent unless the transport explicitly guarantees it (a single ordered queue partition, for example) and the component has verified it depends on only that one ordered source.

Where order matters and cannot be guaranteed by the transport, include a sequence number or timestamp in the payload and have the receiver detect and handle out-of-order and duplicate delivery explicitly, rather than assuming it away.

Distributed transactions and sagas

A single-database transaction gives atomicity for free: either every change commits, or none does. Once a business operation spans more than one service — each with its own database, per TS-5: Application architecture's guidance that each service owns its own state — that guarantee disappears, and the system MUST decide explicitly how it recovers when one step succeeds and a later step fails.

Why two-phase commit is usually the wrong tool

Two-phase commit (2PC) is the classical mechanism for atomic commits across multiple resources: a coordinator asks every participant to prepare, and only commits once every participant has confirmed it can. It gives strong consistency across services, but at a cost that is rarely acceptable in a microservice architecture:

  • Every participant MUST hold locks on the affected rows from the prepare phase until the coordinator’s final decision, blocking other transactions for the duration.
  • If the coordinator fails after the prepare phase but before broadcasting the decision, participants are left blocked indefinitely, holding locks, unable to unilaterally decide whether to commit or abort.
  • It requires every participant to support the same distributed-transaction protocol, which is rarely true across polyglot-persistence services.

2PC SHOULD NOT be the default mechanism for cross-service consistency in a microservice architecture. It is appropriate only for a small, tightly controlled set of participants where its availability cost is acceptable and alternatives genuinely do not fit.

The saga pattern

A saga decomposes a cross-service business operation into a sequence of local transactions, one per service, each of which commits independently. If a step fails, the saga runs compensating transactions for every step that already committed, undoing their effect in reverse order rather than rolling back a single distributed transaction. This trades strong atomicity for availability: no step blocks waiting on the others, and the operation always terminates in a known state (fully applied, or fully compensated) without indefinite locking.

Compensating transactions MUST be idempotent and retryable — see Idempotency and delivery semantics — because they run under the same uncertain-delivery conditions as any other distributed operation, and a saga that cannot safely retry a failed compensation is not actually recoverable. A compensation is not always a literal reversal: "cancel a shipped order" cannot un-ship a package, so its compensation is a new, forward-moving operation (issue a return) rather than a rollback.

Two ways to coordinate the steps of a saga:

  • Choreography. Each service publishes an event when it completes its local transaction; the next service in the sequence reacts to that event and performs its own step. Decentralized — no single point of coordination or failure — but the overall flow is implicit, spread across every participant’s event handlers, which makes a multi-step saga hard to observe or reason about as a whole once it grows past three or four steps.
  • Orchestration. A dedicated orchestrator issues commands to each participant in sequence and tracks the saga’s state explicitly, invoking compensations itself if a step fails. The flow is explicit and centrally observable, at the cost of a new component that itself must be made reliable — its state MUST survive a restart mid-saga, so it is typically backed by durable, replayable storage (an event log or a workflow engine).

Choreography SHOULD be preferred for short sagas (two or three steps) where the coupling it avoids matters more than centralized visibility. Orchestration SHOULD be preferred as saga length or branching complexity grows, because an explicit, inspectable state machine becomes more valuable than avoiding a coordinator once a human has to debug a stuck saga in production.

The dual-write problem and the transactional outbox

A service that writes to its own database and then separately publishes an event describing that write cannot make both operations atomic: if it crashes, or the publish call fails, after the database commit but before the event is published, the write exists but the rest of the system never hears about it. This is the dual-write problem, and it applies to every saga step, not only to sagas — any service that updates its own state and notifies others of that update is exposed to it.

The transactional outbox pattern resolves it. The service writes the event to an "outbox" table in the same local transaction as the business data it describes, so the two either both commit or both roll back — they cannot disagree. A separate process (a poller, or change-data-capture reading the database’s replication log) then reads the outbox table and publishes the event to the message broker, retrying until it succeeds, and marks the row as published only on confirmed delivery. This gives at-least-once delivery of the event (see Idempotency and delivery semantics) with no possibility of a write persisting silently without ever being announced, and without requiring a distributed transaction between the database and the message broker.

Service topology: discovery, gateways, and service mesh

As the number of services grows, the question of how they find and talk to each other stops being an application-code concern and becomes an infrastructure concern in its own right. This section covers that infrastructure layer. It complements TS-49: Cloud platform engineering, which covers the surrounding platform (accounts, environments), and TS-20: Network APIs, which covers the design of an individual API.

Service discovery

A service instance’s network location is not stable — instances are added, removed, and rescheduled continuously (see Fundamentals's fallacy that "topology doesn’t change"). Service discovery is the mechanism by which a caller finds a current, healthy instance of the service it wants to call, rather than a hard-coded address.

  • Client-side discovery. The caller queries a service registry directly and picks an instance itself, applying its own load-balancing logic. Lower latency (no extra hop) but every client MUST implement discovery and balancing logic, or share a library that does.
  • Server-side discovery. The caller sends the request to a well-known address (a load balancer or router), which queries the registry and forwards the request. Simpler for clients, at the cost of an extra network hop and a component that MUST itself be highly available.

Most modern container-orchestration platforms (Kubernetes, and equivalents) provide server-side discovery as a built-in primitive, which is why teams building on such a platform SHOULD default to it rather than implementing either mechanism from scratch.

API gateways

An API gateway is a single entry point that sits between external callers and the internal services that fulfill their requests. It centralizes cross-cutting concerns that would otherwise be duplicated in every externally-facing service: authentication, rate limiting, request routing, TLS termination, and response aggregation across multiple backend calls.

A gateway SHOULD be used at the boundary between the outside world and the system’s internal services — not inserted between every pair of internal services, which just adds a hop and a single point of failure to service-to-service calls that do not need the gateway’s concerns applied to them.

Service mesh

A service mesh extends the same idea — centralizing cross-cutting concerns outside application code — to internal service-to-service traffic. Instead of each service linking a shared library for retries, timeouts, circuit breaking, mutual TLS, and observability (see TS-20: Network APIs and TS-57: Logging, monitoring, observability for those concerns individually), a lightweight proxy is deployed alongside each service instance (the sidecar pattern) and intercepts all traffic to and from it. A control plane configures every sidecar’s behavior centrally.

The mesh’s principal payoff is that cross-cutting behavior can be changed for every service at once, from one place, without touching application code or coordinating a rollout across every team that owns a service. Mutual TLS between every service is the canonical example: enabling it via a shared library requires every service to adopt a new library version on its own schedule; enabling it via a service mesh is a control-plane configuration change that applies mesh-wide.

A service mesh is infrastructure with its own operational cost — an extra proxy hop on every call, and a control plane that is itself a critical dependency of the whole system. It SHOULD be adopted once the number of services and the pain of keeping cross-cutting behavior consistent across them (particularly a shared library that many teams struggle to keep up to date — see Shared libraries at scale) outweighs that operational cost, not as a default starting point for a system with a handful of services.

Microservice contracts

As the number of services grows, an implicit expectation — "every service exposes a health check," "every service emits logs in the standard format" — stops being reliably true unless it is written down and enforced. A microservice contract is an explicit, organization-wide document defining the behaviors every service is required to implement, independent of how each service chooses to implement them. RECOMMENDED minimum contents:

  • Health and readiness endpoints, in a standard shape, that the platform’s orchestrator uses to route traffic and restart unhealthy instances (see Resilience and blast radius).
  • Standard observability output — structured logs, metrics, and traces in the organization’s agreed format, so a new service is observable in the same dashboards as every existing one from day one (see TS-57: Logging, monitoring, observability).
  • Authentication and authorization expectations for both inbound and outbound calls.
  • Graceful shutdown behavior — draining in-flight requests before exiting, rather than dropping them, on receipt of a termination signal.

A service MAY satisfy the contract using a shared library, or by implementing each requirement independently — the contract specifies required behavior, not required implementation, so that polyglot services remain free to choose their own stack while still meeting the same operational bar.

Shared libraries at scale

A shared library that every service links is the natural first way to keep cross-cutting behavior consistent, and it works well at small scale. It stops working as the number of services grows into the hundreds: a breaking change, a security patch, or even a routine version bump now requires coordinating an update across every team that owns an affected service, and some services will always lag. Two structural responses:

  1. Move the concern into infrastructure. Where the concern can be externalized to a proxy or the platform (as with a service mesh, above), do so — infrastructure can be upgraded once, centrally, without any application code change.
  2. Automate the upgrade, not just the release. For behavior that genuinely must live in application code, invest in tooling that opens the update as a change against every consuming repository automatically, rather than relying on every team to notice and act on a new release.

A library-upgrade backlog that grows faster than it is worked down is a signal that the concern it addresses belongs in infrastructure, not in a library every service must remember to update.

Resilience and blast radius

TS-20: Network APIs covers the call-level resilience patterns — timeouts, retries, circuit breakers, rate limiting — that protect one service from one failing dependency. This section covers the system-level counterparts: patterns that limit how far a failure spreads once it occurs, and how a system detects that a component needs to be taken out of rotation in the first place.

Bulkheads

A bulkhead partitions a shared resource — a connection pool, a thread pool, a compute cluster — so that exhaustion in one partition cannot exhaust the resource for every partition. Named for a ship’s watertight compartments: a hull breach floods one compartment, not the whole ship. The most common application is per-dependency resource isolation: a service that calls three downstream dependencies from a single shared connection pool can have that entire pool consumed by one slow dependency, starving calls to the other two that are perfectly healthy. Giving each dependency its own pool, sized to that dependency’s expected load, MUST be considered wherever one slow dependency should not be allowed to degrade calls to unrelated dependencies.

Backpressure

Backpressure is a signal from a consumer to its producer that it cannot keep up with the current rate of work, so the producer SHOULD slow down rather than keep sending at full rate and forcing the consumer to buffer unboundedly, drop work silently, or fall over. A queue with no bound on its depth is not a safety mechanism — it converts an overload problem into a memory-exhaustion problem and delays the point at which anyone notices.

Systems SHOULD apply backpressure explicitly: bounded queues that reject or shed load once full, rather than growing without limit; consumers that pull work at their own sustainable rate rather than having it pushed at them irrespective of their current load; and rate limits on producers, informed by the consumer’s actual observed capacity rather than a value chosen once and never revisited.

Health checks and readiness

A component’s continued participation in a system depends on other components being able to detect, quickly and reliably, that it has become unhealthy. Two distinct checks are RECOMMENDED, because they answer different questions and drive different actions:

  • Liveness. Is the process still running and able to make progress at all? A failed liveness check SHOULD trigger a restart of the instance — the process is assumed to be in a state it cannot recover from on its own.
  • Readiness. Is the instance currently able to serve traffic correctly? An instance can be alive but not ready — still starting up, warming a cache, or temporarily overloaded. A failed readiness check SHOULD remove the instance from load-balancing rotation without restarting it, since the underlying process may recover on its own once the transient condition passes.

Conflating the two — for example, restarting an instance because it is temporarily overloaded — turns a transient, self-recovering condition into an unnecessary restart, which discards any in-flight work and adds cold-start latency exactly when the system can least afford it.

Blast radius and failure domains

A failure domain is the scope within which a single failure is contained — a process, a service, an availability zone, a region. System design SHOULD deliberately choose failure-domain boundaries so that the failure of one component degrades the smallest reasonable portion of the system, rather than cascading across it.

Service boundaries, chosen for the domain-modeling reasons described in TS-5: Application architecture, have a second, independent benefit as failure-domain boundaries: because a microservice is independently deployable and runs in its own process, it can also be given its own runtime stack without that choice affecting any other service. This makes service boundaries a practical place to trial a new language, framework, or runtime under real production traffic — the blast radius of that experiment is contained to the one service, rather than the whole system, so a bad outcome is a contained, reversible decision rather than a system-wide one.

Multi-region and multi-availability-zone deployment extends the same principle to infrastructure failure: distributing instances of a service across independent physical failure domains means the loss of one zone or region degrades capacity rather than causing a total outage. The trade-off is the same latency-versus-consistency cost described in Consistency and availability — keeping state consistent across regions adds cross-region coordination latency to every write that needs it.

Exercising resilience under real conditions

Resilience mechanisms that have never been triggered outside a design document are unproven. Chaos engineering — deliberately injecting failure into a system (killing an instance, adding latency, partitioning a network link) to verify it degrades the way it is designed to — turns an assumption into a tested fact. It SHOULD be practiced regularly, not only after an incident reveals a gap, and it pairs directly with the game-day practice described in TS-57: Logging, monitoring, observability: a game day exercises whether operators can diagnose an injected failure using the available dashboards; chaos engineering exercises whether the system actually recovers from it without operator intervention at all.

Microservices at scale

TS-5: Application architecture covers when and how to extract a service from a monolith. This section covers the concerns that only appear once a system has grown to many services: what must be in place before microservices pay off at all, how to size individual services, and how to keep a large service fleet coherent.

Prerequisites and total cost of ownership

Microservices are not free relative to a monolith — they trade one kind of complexity (a large, tightly coupled codebase) for another (the operational "glue" that holds many independent services together: service discovery, distributed tracing, a deployment pipeline per service, a service mesh or equivalent, cross-service testing). That glue requires sustained investment in infrastructure and tooling, not a one-time setup cost, and it is paid regardless of how simple any individual service is.

Before adopting microservices, an organization SHOULD have, or be committed to building:

  • A deployment pipeline that makes shipping one more service a marginal cost, not a bespoke project — see TS-49: Cloud platform engineering.
  • Observability infrastructure — distributed tracing, centralized logging, per-service dashboards — capable of following a request across service boundaries; see TS-57: Logging, monitoring, observability. Debugging a multi-service request without this is materially harder than debugging a monolith.
  • A team structure that maps to the intended service boundaries, per Conway’s law (see TS-5: Application architecture), since services owned by no one, or by everyone, degrade at the same rate regardless of how well they were designed.

An organization that has not made these investments SHOULD treat that as a signal to stay with, or return to, a modular monolith rather than absorb the distributed-systems complexity described throughout this standard without the infrastructure to manage it. The total cost of ownership of many services — including the cumulative infrastructure overhead of running and monitoring each one, even when individually cheap — SHOULD be weighed explicitly against the coordination cost of the monolith it would replace, not assumed to be lower by default.

Service sizing

Two failure modes exist at opposite ends of the sizing spectrum, and both are common enough to name:

  • Nanoservices. A service is split too small — often along a technical seam rather than a business-capability seam — so that a single logical operation requires calling several services to complete. The distributed- systems overhead described throughout this standard (network calls, partial failure, delivery semantics) is now paid for a decomposition that bought no real independence in return, since the resulting services still change together. A service SHOULD encapsulate a complete business capability, per the bounded-context guidance in TS-5: Application architecture — not a single database table, or a single technical operation, split out for its own sake.
  • Overgrown services. A service that started well-scoped accretes unrelated responsibilities over time until it needs multiple teams to operate it safely, recreating monolith-style coordination costs inside a single service. This is the signal to split it — along the same bounded-context reasoning used to decide the original boundary, not along whatever line is most convenient to extract first.

Neither failure mode is purely a one-time decision — a system’s ideal service sizing shifts as its domain and organization change, so service boundaries SHOULD be revisited periodically rather than treated as permanent once drawn.

A large number of services also carries a cumulative cost independent of any one service’s size: each one carries its own baseline infrastructure overhead (a deployment pipeline, monitoring, minimum resource allocation), and that overhead compounds across hundreds or thousands of services even when each is individually well-sized. This cumulative cost is part of the total cost of ownership described above, and SHOULD factor into the decision to split a service, not just the service’s own internal coherence.

SOLID at the service level

The SOLID principles, usually applied to classes and modules, generalize directly to service boundaries:

  • Single responsibility. A service SHOULD own one business capability, not several unrelated ones — the service-level restatement of the nanoservice and overgrown-service guidance above.
  • Open/closed. A service SHOULD be extensible by its consumers without requiring changes to the service itself — through versioned APIs, feature flags, or explicit extension points — rather than requiring every new consumer need to be accommodated by modifying the service’s internals.
  • Liskov substitution. Where more than one service implements the same contract (multiple regional deployments of the same service, or a new implementation replacing an old one behind a stable interface), any implementation MUST be substitutable for another without breaking callers that depend only on the contract.
  • Interface segregation. A service’s public, external-facing API SHOULD be distinct from its internal, trusted-caller API, so that external consumers are not exposed to — and cannot come to depend on — operations meant only for internal use.
  • Dependency inversion. Services SHOULD depend on abstractions — message contracts published to a broker, or a versioned API contract — rather than on each other’s concrete implementations. Routing service-to-service communication through a message broker (see TS-23: Messages and events) is the service-level equivalent of depending on an interface rather than a concrete class: publishers do not know, or need to know, which services will consume what they publish.

Monolith-to-microservices migration execution

TS-5: Application architecture covers when service extraction is premature. This section covers how to execute an extraction once a component’s interface has stabilized and the prerequisites in Microservices at scale are in place. It complements TS-45: Data migrations, which covers the mechanics of moving the underlying data — extracting a service almost always requires migrating the data it owns, and that data migration SHOULD follow TS-45’s expand-and-contract discipline in full.

The strangler fig pattern

Replacing a monolith with a big-bang rewrite carries the highest risk of any migration strategy: the new system is unproven until the moment it fully replaces the old one, by which point the cost of discovering a fundamental problem is highest. The strangler fig pattern, named for a vine that gradually envelops and eventually replaces its host tree, avoids this by routing an increasing share of traffic through new services while the monolith continues serving everything not yet migrated:

  1. Introduce a routing layer (a proxy, gateway, or feature flag) in front of the monolith, capable of directing individual requests to either the monolith or a new service.
  2. Extract one bounded piece of functionality into a new service.
  3. Redirect the routing layer to send matching requests to the new service instead of the monolith.
  4. Repeat for the next piece, until the monolith serves nothing that has not already been extracted, at which point it can be retired.

Each step is small, independently reversible (route back to the monolith if the new service misbehaves), and delivers value as it lands rather than only once the whole migration completes. This is the same incremental, independently-reversible discipline as TS-45’s expand-and-contract pattern, applied to service boundaries instead of schema changes.

Prioritizing what to extract first

A large migration cannot extract everything at once, and the order of extraction materially affects the risk and value delivered along the way. Two factors SHOULD drive prioritization:

  • Boundary stability. Extract components whose interface and domain model have already stabilized before components still under active redesign — extracting an unstable boundary just relocates the churn into a more expensive, cross-service form, per the premature-decomposition guidance in TS-5: Application architecture.
  • Isolation. Extract components with fewer dependencies on the rest of the monolith before deeply entangled ones. Early extractions validate the migration approach — routing, data ownership, operational tooling — while the cost of getting it wrong is still low.

A migration plan SHOULD sequence extractions explicitly, in writing, rather than extracting opportunistically — see TS-3: Design docs for how to capture that plan.

Parallel running

For a piece of functionality where correctness matters enough that a routing switch alone is not sufficient evidence of success — a pricing engine, a billing calculation — run the new service and the monolith’s equivalent code path side by side, compare their outputs on real traffic, and only cut over once they agree consistently. This is the service-extraction counterpart of TS-45’s shadow-mode dual-write verification, applied to behavior rather than to stored data: the monolith continues to serve the response, while the new service’s output is logged and diffed against it silently, so that disagreements are caught before they can ever reach a user.

Sequencing over a multi-year timeline

A large monolith-to-microservices migration commonly spans years, not sprints, and MUST be planned as a long-running, prioritized program rather than a single project with a fixed end date. Concretely, this means:

  • The extraction backlog SHOULD be re-prioritized periodically as the system and organization change, rather than fixed once at the start — the reasoning that made a given boundary the right next extraction two years ago may no longer hold.
  • Business and engineering leadership SHOULD agree explicitly on the technical investment a multi-year migration requires, since a migration that competes for every sprint against feature work will stall indefinitely without a standing allocation of capacity.
  • The monolith and the extracted services will coexist for a long period, not briefly — so the monolith SHOULD continue to receive the same operational care (monitoring, dependency updates, incident response) as the services replacing it, for as long as it is still handling live traffic.

Continuous technology evaluation

TS-5: Application architecture covers choosing a framework for a single application. At the scale of a distributed system — many services, potentially many teams, evolving over years — the technology choices behind those services are not a one-time decision made at each service’s inception. They are a standing commitment that SHOULD be revisited on an ongoing basis, at the level of the whole system’s technology strategy, not just service by service.

Triggers for adopting new technology

Adopting new technology has a real cost — migration effort, a new set of operational failure modes to learn, another item in the system’s total technology surface area — so "slightly better than what we have" SHOULD NOT be sufficient justification on its own. Adoption is justified once an existing choice produces a concrete, current problem, typically one of:

  • Cost. The current technology’s cost has grown disproportionately to the value it delivers, discovered through regular cost review rather than only when a bill spikes unexpectedly.
  • Scaling limits. The current technology can no longer meet the system’s actual load or growth trajectory, evidenced by measured limits, not speculation about future limits that may never be reached.
  • A genuine new requirement. A customer or product need exists that the current stack cannot reasonably meet.

Build versus buy

Once a real problem justifies a change, evaluate build-versus-buy with a short, real-world test rather than a lengthy paper evaluation: run a proof-of-concept against production-representative data and load, and measure the result against the actual problem, before committing. A vendor’s stated pricing and performance figures SHOULD be verified against the system’s own workload before being relied on for a decision — published figures routinely diverge from the cost or performance a specific workload actually experiences.

Combined evaluation criteria

Technology decisions SHOULD be evaluated against a criteria set that deliberately mixes technical and business factors, rather than treating them as two separate, sequential reviews:

  • Performance and scalability against the system’s actual, current workload.
  • Cost, including the operational cost of running and maintaining the technology, not only its licensing or infrastructure price.
  • Reliability, including the technology’s own failure modes and how they interact with the resilience patterns described in Resilience and blast radius.
  • Support and community maturity — an actively maintained project with a responsive community or vendor materially reduces the operational risk of adoption, independent of the technology’s technical merits.
  • Flexibility and interoperability with the rest of the system’s existing stack, since a technology that fits everywhere else eases every future integration, while one that fits nowhere else becomes an island that someone eventually has to maintain alone.

Ongoing re-evaluation

A technology decision is not permanent once made. Even a technology the system has depended on successfully for years SHOULD be periodically re-evaluated against newer alternatives using the same criteria that justified adopting it in the first place — not out of restlessness, but because the triggers in Triggers for adopting new technology can appear gradually, and a scheduled review catches them before they become an emergency. This is distinct from, and does not replace, the ordinary dependency-maintenance discipline described in TS-5: Application architecture — re-evaluation asks whether the technology itself is still the right choice, not just whether its current version is current.

Organizations operating many services over a multi-year horizon SHOULD maintain an explicit, living view of their technology strategy for system-wide capabilities — the messaging infrastructure, the primary datastore technologies, the service mesh — so that individual service teams' technology choices accumulate toward a coherent direction rather than diverging independently. This does not mean centrally mandating every technology choice (see the autonomy-versus-alignment guidance in TS-49: Cloud platform engineering) — it means the organization has a stated, current answer to "where is our technology stack heading," so that individual choices can be evaluated against it rather than made in a vacuum.

References