Observability

Observability is both a quality attribute of complex systems and a set of strategies and tools for understanding and managing them. It is a subset of visibility and complements traditional monitoring strategies.

As a quality attribute, a system’s observability can be improved in many ways: adding more logs and metrics, or training a team to use monitoring tools.

But more commonly we talk about observability as a set of tools and strategies that let us ask questions about our systems – questions that, perhaps, we didn’t know we needed to ask from outside the system. With observability tools, you can discover the questions to ask by exploring your system’s telemetry data.

This is done by analyzing telemetry data to track how the internal state of a system changes over time and to diagnose unexpected behaviors. The goal is to understand how and why a system behaves as it does, supporting deeper analysis and troubleshooting.

While monitoring helps us to deal with the known unknowns, observability helps us to discover the unknown unknowns.

Observability strategies originated in distributed software design. In complex systems like distributed ones, failures often result from complex interactions between components rather than from isolated events. Distributed software fails in new and unexpected ways. You are often dealing with "unknown unknowns", because the system is constantly evolving, with different teams incrementing different components at different times. It is impossible to predict every way that such systems might fail.

In such complex systems, it is necessary to collect and analyze data from multiple sources to gain a complete view of overall system behavior.

Observability has been described as the business intelligence of developer tools. It encompasses three types of tools – the three pillars of observability:

  • Logging: Recording discrete events or states.
  • Metrics: Aggregating numerical data points over time, and monitoring changing patterns in those metrics.
  • Tracing: Tracking the flow of requests through the system.

Observability strategies may emphasize one or more of these pillars, depending on the specific needs of the system. Metrics are ideal for recording event occurrences, item counts, action durations, or reporting the status of resources like CPU and memory. Logs are more suited to capturing detailed narratives of events, especially for errors and warnings. Traces offer a window into the journey of individual request-response cycles, and are particularly beneficial to track changing state through asynchronous messaging systems.

In a well-understood system, it may be sufficient to prioritize high quality metrics over detailed (and expensive) logs and complete traces. But the more complex the system, the more difficult it is to anticipate all the information you will need, and so to define the required metrics up-front. Visibility of such systems therefore leans more heavily on logs and traces, and less on monitoring and metrics.

Tracing addresses the difficulties of monitoring and debugging distributed software. It helps us understand cause and effect in complex distributed systems: what effect does a component publishing an event have on the rest of the system, and how do you understand what happened upstream?

For this reason, the most important of the three pillars in a distributed, asynchronous system is distributed tracing.

In most use cases, some combination of all three strategies – metrics, logging, and tracing – will be required. Add in traditional monitoring strategies such as application performance management, and you will have comprehensive visibility of the internal workings of a system. For example, you should be able to detect problems through metrics, diagnose them through logs, and use traces to understand the flow of requests that led to the issue. Without a combination of all three strategies, you may miss important context or details that help you to understand the underlying causes of issues.

The relative balance between the three strategies – metrics, logging, and tracing – will change over time, and they will each inform and shape the others. For example, imagine that a business-critical database, in the order processing services, goes offline. You diagnose the issue using logs, and discover that the database has run out of disk space. You fix the issue and restart the database. Then you put in place monitoring and alerting on metrics for disk usage, so you can proactively resolve the issue before it happens next time.

Besides the trade-offs between the three pillars of observability, there are also trade-offs to be made in the depth and breadth of the telemetry data collected. More data will provide greater insights, but with additional cost for storage and processing. In large-scale systems, you may not be able to afford capturing every piece of available data, else you could end up with a US$65m bill. You need to balance the cost of collecting and analyzing telemetry against the value it provides.

The best approach, usually, is to start with a basic set of metrics, logs, and traces, and then iterate as you learn more about your system. This iterative approach helps you refine your strategy over time and collect the most valuable data for understanding and optimizing your system.

Observability tools

Related links