TS-46: Distributed Data and Caching

This technical standard covers best practices for distributed data and caching in software systems. This is general guidance, not specific to any particular technology or application layer.

Distributed data is data that is spread across, or replicated between, multiple locations – different services, databases, processes, or machines – rather than living in a single authoritative store. We distribute data to make systems faster, more available, and more scalable. But once the same data exists in more than one place, we inherit a new class of problems: keeping copies consistent, deciding which copy is authoritative, and reconciling concurrent changes.

Caching is one objective of distributed data. A cache is a deliberately placed copy of data, positioned closer to where it’s needed so it can be read faster or more cheaply than fetching it from its source. Like all distributed data, caching trades single-source simplicity for performance – and inherits the consistency problems that come with it. Most caching bugs don’t come from Redis or Memcached, or whatever technology is used to implement the cache – they come from bad cache design.

Caching tradeoffs

Caching is the most common form of distributed data, so we begin there. The same tradeoffs apply, in different proportions, to any data we choose to distribute.

There are lots of costs to caching, including:

  • Harder debugging and code profiling.
  • Users see outdated information.
  • Increased memory footprint (because you just move resource utilization, you don’t actually reduce it).
  • Increased system complexity.

We must have clear reasons to implement caching layers in our applications, and these must be clearly defined and documented.

Factors for caching

Before implementing a new caching layer, consider the following. These factors decide whether a given piece of data is a good candidate for caching specifically – distributing it as a read-optimized copy.

Data access frequency

If a piece of data is rarely accessed, it might not benefit from being cached. But if it’s hit constantly, whether by the same user or across many clients, then it may be worth investigating the pros and cons of caching it.

Cost of data access

Is it expensive to retrieve the data?

Not all reads are equal. Some hit external services, multiple databases, join several tables, and compute summaries that cost real CPU. For example, if you’re generating a user’s analytics dashboard, this may involve multiple service calls and heavy aggregation. It may make sense to cache the final computed payload.

But fetching a flat record by a primary key from a well-indexed table – the cost of doing that lookup will already be minimal.

Data stability

Is the data stable or volatile?

Stable data makes a great cache. It can sit there for a long time without anyone noticing.

But people notice when volatile data (ie. data that gets stale quickly) is cached for too long.

Therefore, particular consideration needs to be given to cache invalidation for volatile data. (See below for more discussion on this topic.)

Data size and complexity

How big is the data object? Is it cheap to serialize and deserialize?

Big, messy data doesn’t belong in fast memory. Large payloads eat up space, increase pressure on garbage collection systems, and slow down serialization and deserialization.

Small, flat data is faster to work with and easier to evict if needed.

A classic antipattern is caching an entire product catalog (~100K items) instead of caching paginated views or product summaries.

Caching is a fast-access shelf, not cold storage. Store what fits – what you’ll grab often and quickly.

Caching is kept fast by storing simple shapes against small keys.

Impact on user experience

Not all latency matters. But when it does, it tends to matter a lot.

Anything on the critical path of a user interaction – loading a web page, rendering a component, or hitting "submit" – should feel instant. If caching makes that possible, use it.

For example, if the response time for a search query directly impacts conversion, this is a good candidate for caching. But a background sync task at 2AM - probably not.

Safeness

Is it safe to cache?

Fast is good. Leaky is not.

Caching user-specific or sensitive data – PII, tokens, financials – without scoping or encryption is a security risk. One bad key and someone sees what they shouldn’t.

For example, a shared cache key for a user profile might accidentally leak another user’s data in a multi-tenant app.

To mitigate, use per-user or per-session cache keys; encrypt values where possible; and set short TTLs for sensitive data.

As a rule of thumb, if it can’t go in a log file, it probably shouldn’t go in a cache either.

Scalability

Will the cache scale?

Caching that works for 1,000 users can still collapse for 1 million. Unbounded keys, high churn, or poorly managed TTLs can overwhelm memory, reduce hit ratios, and cause eviction storms.

Tactics: use eviction policies (LRU, LFU); set size limits; monitor hit/miss ratio and eviction churn.

Evaluating a cache

To decide whether to cache a given piece of data, in summary:

  • Cache what’s used frequently.
  • Cache what’s expensive to fetch.
  • Cache what stays valid for long enough to be worth it.
  • Cache what improves something a user can actually feel.
  • Don’t cache what you can’t keep safe or what won’t scale.

For a rough calculation of the value of caching a piece of data:

cache value = access frequency x retrieval cost x stability

If any of these – access frequency, retrieval cost, or data stability – are near zero, then caching that data won’t give you much back. But if all three are high, you’re probably sitting on a high-leverage cache opportunity.

If you can’t explain why something is in a cache, it probably shouldn’t be. Every cache key is a liability, so it must justify its existence by delivery value.

We should design each cache with the same intent we would a database schema or API contract. Caches are a deliberate, visible part of the architecture, with trade-offs, constraints, and clear justification.

Caching works best when it’s boring, predictable, scoped, and justified.

Cache freshness

Keeping copies consistent is the central problem of all distributed data. For a cache – a read-optimized copy – that problem takes a specific form: freshness. Once you’ve decided to cache something, the next question is: how do you keep it fresh?

You have two main options:

  • Time-to-Live (TTL): This is a configuration that expires the cache after a fixed time.
  • Invalidation: You explicitly remove or update the cache when the data changes.

Each option has trade-offs. As a guide:

  • Stable data + frequent reads → use TTL
  • Stable data + infrequent reads → avoid TTL (wasteful)
  • Volatile data + frequent reads → use invalidation
  • Volatile data + infrequent reads → avoid caching altogether

Distributed writes

Distributed writes mean several clients may try to update the same data at the same time. There are three ways databases can be designed to handle this:

  1. Write-ahead logging (WAL): Used by PostgreSQL and MySQL. The change is written ahead to a log, before being applied to the database. This approach is safe and durable, but it doesn’t properly handle concurrent write conflicts, because updates are simply applied in the order in which the write logs are written. Another downside of this design is the overhead of monitoring the logs. The main upside, however, is that it is non-blocking – no ongoing write will block another one from being performed.
  2. Locking (aka. pessimistic concurrency): Used by traditional RDBMS. The writer acquires a lock on the record and, while that lock is in place, other writers are blocked until the lock is released. The initial writer update the data and releases the lock. This prevents conflicting updates, but it does not scale well because of blocking operations – so it’s a poor fit for high-throughput distributed systems. The main downside are risk of deadlocks, high latency, and poor scalability. But the upsides are strong consistency, and the system is easy to reason about.
  3. Versioning (aka. optimistic concurrency): Used by Dynamo DB, Cassandra, and the Event Sourcing pattern. Locks are replaced with version checks. Then client reads version N, and the client then writes with condition "version = n"
    • known as a conditional write. The write succeeds only if the version still matches. Writes may still fail, and the client is responsible for dealing with merge conflicts and data retries, but at least there’s no blocking of new connections, so this scales very well and supports high throughput. Clients assume no conflicts by default, but handle them gracefully when they occur. This prevents silent overwrites - only one writer succeeds per version, eliminating the "last write wins" chaos. The main downsides are on the client-side: conflicts are made visible for clients, who require more logic to deal with them. The trade-offs are there’s no lock contention, and there’s no centralized locking so it scales very well.