Observability

Observability is both a quality attribute of complex systems and a set of strategies and tools for understanding and managing them. It is a subset of visibility and complements traditional monitoring strategies.

As a quality attribute, a system’s observability can be improved in many ways: adding more logs and metrics, or training a team to use monitoring tools.

But more commonly we talk about observability as a set of tools and strategies that let us ask questions about our systems – questions that, perhaps, we didn’t know we needed to ask from outside the system. With observability tools, you can discover the questions to ask by exploring your system’s telemetry data.

This is done by analyzing telemetry data to track how the internal state of a system changes over time and to diagnose unexpected behaviors. The goal is to understand how and why a system behaves as it does, supporting deeper analysis and troubleshooting.

While monitoring helps us to deal with the known unknowns, observability helps us to discover the unknown unknowns.

Observability strategies originated in distributed software design. In complex systems like distributed ones, failures often result from complex interactions between components rather than from isolated events. Distributed software fails in new and unexpected ways. You are often dealing with "unknown unknowns", because the system is constantly evolving, with different teams incrementing different components at different times. It is impossible to predict every way that such systems might fail.

In such complex systems, it is necessary to collect and analyze data from multiple sources to gain a complete view of overall system behavior.

Observability has been described as the business intelligence of developer tools. It encompasses three types of tools – the three pillars of observability:

  • Logging: Recording discrete events or states.
  • Metrics: Aggregating numerical data points over time, and monitoring changing patterns in those metrics.
  • Tracing: Tracking the flow of requests through the system.

Observability strategies may emphasize one or more of these pillars, depending on the specific needs of the system. Metrics are ideal for recording event occurrences, item counts, action durations, or reporting the status of resources like CPU and memory. Logs are more suited to capturing detailed narratives of events, especially for errors and warnings. Traces offer a window into the journey of individual request-response cycles, and are particularly beneficial to track changing state through asynchronous messaging systems. Many teams now add continuous profiling as a fourth signal, attributing resource costs to specific code paths in production.

Logs are most useful when they are structured, eg. JSON with consistent fields, rather than free text.

In a well-understood system, it may be sufficient to prioritize high quality metrics over detailed (and expensive) logs and complete traces. But the more complex the system, the more difficult it is to anticipate all the information you will need, and so to define the required metrics up-front. Visibility of such systems therefore leans more heavily on logs and traces, and less on monitoring and metrics.

The pillars are most powerful when they are linked. For example, putting the trace ID in every log line lets you jump from a slow trace to the log lines that explain it. Context propagation is what carries those IDs across service boundaries, typically using the W3C Trace Context HTTP headers.

Tracing addresses the difficulties of monitoring and debugging distributed software. It helps us understand cause and effect in complex distributed systems: what effect does a component publishing an event have on the rest of the system, and how do you understand what happened upstream?

For this reason, the most important of the three pillars in a distributed, asynchronous system is distributed tracing.

In most use cases, some combination of all three strategies – metrics, logging, and tracing – will be required. Add in traditional monitoring strategies such as application performance management, and you will have comprehensive visibility of the internal workings of a system. For example, you should be able to detect problems through metrics, diagnose them through logs, and use traces to understand the flow of requests that led to the issue. Without a combination of all three strategies, you may miss important context or details that help you to understand the underlying causes of issues.

The relative balance between the three strategies – metrics, logging, and tracing – will change over time, and they will each inform and shape the others. For example, imagine that a business-critical database, in the order processing services, goes offline. You diagnose the issue using logs, and discover that the database has run out of disk space. You fix the issue and restart the database. Then you put in place monitoring and alerting on metrics for disk usage, so you can proactively resolve the issue before it happens next time.

Besides the trade-offs between the three pillars of observability, there are also trade-offs to be made in the depth and breadth of the telemetry data collected. More data will provide greater insights, but with additional cost for storage and processing. In large-scale systems, you may not be able to afford capturing every piece of available data, else you could end up with a US$65m bill. You need to balance the cost of collecting and analyzing telemetry against the value it provides.

Trace sampling is the usual way to control volume, and comes in two forms. Head-based sampling decides whether to keep a trace at the start of the request, which is cheap but blind to the outcome. Tail-based sampling decides after the trace completes, so it can keep every error and slow request, at the cost of buffering spans until then. High-cardinality values such as user IDs and request IDs belong in logs and traces, not in metric labels, where they explode storage costs. See cardinality.

The best approach, usually, is to start with a basic set of metrics, logs, and traces, and then iterate as you learn more about your system. This iterative approach helps you refine your strategy over time and collect the most valuable data for understanding and optimizing your system.

Deciding what to measure

A few frameworks help to choose a starting set of signals.

  • RED method. Rate, Errors, and Duration. Suits request-driven services.
  • USE method. Utilization, Saturation, and Errors. Suits resources such as CPUs and disks.
  • Four golden signals. Google’s set of latency, traffic, errors, and saturation.

Measurements such as these become the indicators behind service level objectives, and are best alerted on by error budget burn rate. See alerting.

Observability tools

  • OpenTelemetry: The vendor-neutral standard for instrumenting and exporting all signals.
  • Prometheus and Grafana: Common open source choices for metrics and dashboards.
  • Loki: Log aggregation, which pairs with Grafana.
  • Jaeger and Tempo: Distributed tracing backends.
  • Datadog and New Relic: Commercial observability platforms.
  • Elastic: The vendor behind the Elastic stack, a suite of tools for search, analytics, and observability.
  • Honeycomb: Observability service.
  • Splunk: Now owned by Cisco.

Related links

See also