TS-57: Logging, monitoring, observability

This technical standard sets out principles and best practices for logging, monitoring, and observability – collectively known as the visibility quality attribute.

Logging

Personal Identifiable Information (PII) MUST NOT be sent to log output.

Runtime errors MUST be logged, so that issues can be detected and investigated.

Monitoring

Uptime monitoring MUST be implemented for software-as-a-service applications. Individual services within a distributed system MUST be monitored separately to the user interface of the application as a whole.

Alerting

Production alerting – notifications of errors or other issues – MUST be implemented in production environments, and SHOULD be implemented in pre-production environments.

Oncall health

Alerting exists to reach a human, and the oncall rotation that receives those alerts is a system with its own failure modes — alert fatigue, excessive page volume, and burnout — that degrades the effectiveness of every other observability investment if left unmanaged. Oncall health SHOULD be measured and tracked with the same rigor as any other production metric, using signals such as:

  • Page volume per rotation and per engineer, including a breakdown of actionable versus non-actionable pages.
  • Time-of-day distribution of pages, since a page load that is otherwise acceptable becomes a wellbeing problem when it concentrates overnight.
  • Time to resolution, and the proportion of pages that resolve without human intervention once acknowledged.

Where these signals show an oncall rotation is unhealthy, fixing it SHOULD be prioritized over product work. An engineering organization that lets an unhealthy oncall rotation persist while continuing to ship features is trading a visible, short-term output for an invisible, compounding cost: alert fatigue causes real alerts to be missed, and burnout drives the engineers who best understand the system to leave. Remediation includes tuning alert thresholds to cut non-actionable pages, fixing the underlying causes of recurring pages rather than re-paging on them indefinitely, and adjusting rotation size or shift length where the volume of a single rotation is unsustainable.

Metrics

Metrics MUST be gathered to inform and evaluate:

  • Success criteria for changes made.
  • Future development.
  • Stability and reliability monitoring.
  • Customer usage / activity patterns.

Key metrics – eg. API response times, batch processing times, speed of transactions etc. – MUST be tracked and monitored.

Customer-facing applications MUST have front-end analytics integrated, to provide insights into how customers use the application.

Observability strategy

Observability is the degree to which a system’s internal state can be inferred from its external outputs. Logging, monitoring, alerting, and metrics are the mechanisms; observability is the outcome — the ability to ask arbitrary questions of a running system without having to instrument it anew for each question.

A system with extensive telemetry but poor structure is not observable. The value of observability comes from treating it as a product: designed for the engineers who use it, consistent across systems, and exercised before it is needed.

Treat observability as a product

Observability tooling is used under pressure — during incidents, when the cost of confusion is highest. It SHOULD be designed with the same care as a user-facing product:

  • Usability. Dashboards and logs should be easy to find, read, and navigate. An engineer who has never seen a particular dashboard SHOULD be able to orient themselves within seconds.
  • Consistency. The structure of dashboards and logs SHOULD be consistent across all systems, so familiarity with one transfers to all others.
  • Ownership. Observability is an engineering-team-wide responsibility, not the specialty of a few infrastructure engineers. Every team that owns a service owns its dashboards, logs, and alerts.

Ownership requires self-service tooling to be meaningful. Where changing a dashboard, adding a metric, or configuring an alert requires a ticket to a centralized observability or platform team, ownership sits with that team in practice, whatever the stated policy says — and a request routed through a queue turns a same-day investigation into a week-long wait. Development teams MUST be able to configure their own dashboards, alerts, and metrics directly, without going through a centralized team. A centralized team MAY still own the underlying platform — provisioning, retention policy, cost control — but MUST NOT be the sole route through which a development team’s own observability configuration changes.

Structure dashboards as a hierarchy

Dashboards SHOULD be organized as a drill-down hierarchy that lets an engineer move from "something is wrong" to "this specific request failed" in a few clicks. A RECOMMENDED hierarchy has four levels:

  1. Overview dashboard — A single, bird’s-eye view of every subsystem, acting as a traffic light. Each subsystem’s health is summarized by a small number of high-level signals that go red when anything is wrong. This is the first stop when an alert fires: it points to which subsystem is unhappy. It SHOULD be the default landing page for the dashboard tool and SHOULD be linked from team channels.
  2. System dashboards — One per subsystem, giving an exhaustive picture of that subsystem’s health. A system dashboard SHOULD answer: where are requests being rejected, where are the bottlenecks, what are the outcomes, and how much strain is the system under? Each system dashboard SHOULD link out to the logs for that subsystem.
  3. Logs — The individual events. See Logging for log design.
  4. Traces — The most zoomed-in view, showing where a single request spent its time. Traces are the tool of last resort for slow requests.

Each level SHOULD link to the next, so an engineer moves down the hierarchy without leaving the dashboard. Moving back up — from a trace to the broader context — SHOULD be equally easy.

Dashboard design

  • Optimize for a glance. An engineer SHOULD be able to tell whether a dashboard is relevant to their investigation at a glance. Each dashboard SHOULD have a clear focus and direct the reader to where they need to go next.
  • Avoid false negatives. An overview dashboard SHOULD go red when anything is wrong, no matter how minor. It is better to investigate a false alarm than to miss a real problem because the dashboard looked green.
  • Use a consistent design system. Dashboards created by different teams SHOULD follow the same layout, color, and panel conventions, so an engineer comfortable with one dashboard can transfer that familiarity to any other.
  • Split metrics by outcome. Where a request can have several outcomes (eg. success, rate_limited, error), metrics SHOULD be split by outcome so the dashboard shows not just how many requests there were, but how they were handled.

Event logs

Each significant unit of work SHOULD emit a single, consistently formatted event log on completion, regardless of outcome (success, error, or panic). An event log records the event name, duration, outcome, and the IDs of the resources involved — the high-cardinality data that cannot be captured in metrics.

Event logs bridge the gap between metrics and logs: a metric tells you that something is wrong; the event log gives you a specific example of what went wrong. Event logs SHOULD share labels with the corresponding metrics so an engineer can move from a chart to a log line without changing query.

Note

Emit the event log in a defer block (or equivalent) so it is always written, even when the operation panics or returns early.

Exemplars

Where a metric is tracked, an exemplar — a reference to a specific request that contributed to that metric value — MAY be attached. Exemplars let an engineer jump from a point on a chart directly to the trace for the request that produced it, collapsing the metric-to-trace step into a single click.

Tracing

Traces show where a single request spent its time. They are the tool of last resort for investigating slow requests. To make traces useful:

  • Trace third-party interactions. Every call to an external API SHOULD be traced, so a slow request is immediately attributable to an upstream dependency. Use a shared base client that applies tracing as standard.
  • Trace database queries. Each query SHOULD be traced with its duration and, where practical, the query itself, so an engineer can see both the slow query and an example of it. Time spent waiting for a connection SHOULD be tracked separately from time spent executing the query, to distinguish contention from inefficiency.

Debugging as a discipline

Observability supplies the raw material — logs, traces, metrics, dumps — but turning that material into a root cause is a separate skill: forming a hypothesis, finding the observation that would confirm or refute it, and repeating until the cause is isolated. Treat debugging as a discipline to practice, not an ability engineers either have or lack:

  • Reason across layers. The hardest bugs span multiple abstraction layers — application code, runtime, kernel, network, hardware — and root-causing them requires moving between layers rather than staying at the one where the symptom appeared. An engineer with a strong mental model of the full stack can sometimes single-shot a bug from one observation, by reasoning through what system state would produce it; an engineer without that model has to fall back to trial and error.
  • Form a hypothesis before gathering more data. Undirected log-scrolling is slow. State what you expect to be true if the hypothesis holds, find the cheapest observation that would confirm or refute it, and only then look.
  • Prefer the observation that eliminates the most hypotheses. Where several observations would each move the investigation forward, the one that splits the remaining hypothesis space roughly in half is worth more than one that only confirms the leading theory.

When to stop deducing and start observing

Deductive root-causing does not scale to every system. For a distributed system, a codebase with years of undocumented accretion (a "big ball of mud"), or heterogeneous client-side code running on devices you do not control, building a mental model precise enough to reason from is often impractical — there are too many interacting components, too much unobserved state, or too little control over the runtime environment.

Where deductive understanding is impractical, switch to an empirical strategy instead:

  • Treat behavior statistically. Instead of explaining any one failure, characterize the failure rate, its distribution across inputs or instances, and how it moves in response to changes. A statistical answer ("errors correlate with payload size above 1 MB") is often actionable even where a full causal explanation is not.
  • Invest in fault tolerance over root-causing every failure. In a distributed system, some component failures are not worth explaining individually — retries, circuit breakers, and graceful degradation absorb them more cheaply than root-causing each one. Root-cause the failures that recur or that the fault-tolerance layer does not absorb; let the fault-tolerance layer handle the rest. See TS-6: Distributed system design for the resilience patterns this relies on.

This is a judgment call, not a rule with a fixed threshold: attempt deductive root-causing first, and switch to empirical methods once the cost of building an accurate mental model clearly exceeds the cost of living with a statistical answer.

Exercise the setup

An observability setup that has never been used under pressure is unproven. Game days — scheduled exercises in which the team deliberately breaks something and uses the dashboards to diagnose it — SHOULD be run regularly. Game days both validate that the dashboards work and build the team’s familiarity with them, so that real incidents are not the first time anyone opens the tooling.

Treat game days like user testing: give participants minimal context, watch what they find easy and what they struggle with, and adjust the dashboards together afterward.

References