Data retention
Data retention is the quality attribute and associated policy that governs how long a system keeps data before deleting or archiving it. A retention policy specifies, for each class of data, how long it must be held, where it is stored, and what happens to it at the end of that period.
Retention is driven by three forces that often pull in different directions. Regulatory and compliance obligations frequently mandate minimum retention periods, eg. financial records that must be kept for seven years, or audit logs that must survive for a defined window. Business value pulls the other way. Historical data feeds analytics, trend detection, and machine-learning models, so there is often pressure to keep data longer than the law strictly requires. Privacy considerations and data-minimization principles push toward shorter retention, since holding data longer than necessary increases the cost and blast radius of a breach and may conflict with legal rights such as the right to erasure.
The third force is cost and operability. Every byte retained consumes storage, backup bandwidth, and compute at query time. Observability pipelines in particular can generate prodigious volumes of telemetry and log data, and unbounded retention is a common source of runaway cloud bills. A retention policy is the mechanism that keeps these forces in balance.
Trade-offs and pitfalls
Deleting data sounds simple, but in practice it is one of the harder problems in data engineering. A few common pitfalls.
- Immutable and append-only stores. Event-sourced
systems and change data capture pipelines
record facts that, by design, cannot be altered or removed. Honouring a
deletion request then requires compensating events, tombstones, or
key-level encryption rotation rather than a simple
DELETE. - Backups and replicas. A record purged from the primary store typically lives on in disaster recovery backups, snapshots, and replicas for as long as those copies are retained. True deletion has to account for the retention window of every copy, or the policy is only half-enforced.
- Tombstones vs. physical deletion. Many systems implement retention by marking records as deleted (a tombstone) and physically removing them later, in a batch. The window between logical and physical deletion must itself be part of the policy.
- Retention vs. the right to erasure. A legal requirement to keep data can conflict with a user’s right to have it deleted. Reconciling the two usually means redacting or anonymizing identifying fields while retaining the record itself for its mandated period.
Retention also interacts with data integrity and durability. A policy is only meaningful if the data it governs remains correct and recoverable for as long as it is held, and is reliably removed once that period elapses.