A checkout dashboard turns red, but every service reports healthy CPU and memory. The payment request was slow because an inventory lookup retried behind a congested dependency, then exhausted its timeout budget. The dashboard detected a symptom. Understanding the path requires evidence about how the request moved through the system.
This is the practical distinction behind observability vs monitoring. Monitoring checks conditions you have chosen to watch. Observability describes the ability to infer a system’s internal behavior from its outputs. Modern teams need both: dependable alerts for known risks and enough connected telemetry to investigate unfamiliar failures.
Distributed tracing contributes the missing request-level narrative. It shows which operations participated, their timing, and their relationships. Used with metrics, structured logs, and domain knowledge, a trace can turn “checkout is slow” into an explanation that engineers can test.
Monitoring and observability answer related questions
Monitoring asks whether a measured condition crosses an expected boundary: error rate, queue age, available capacity, or successful purchase rate. It is especially useful when the team knows what failure looks like and which action should follow. A clear alert tied to an owner is often more valuable than a visually elaborate dashboard.
Observability supports questions that were not fully anticipated when instrumentation was written. Which release introduced the problem? Does it affect one region or one customer workflow? Are successful requests slow too? The answer depends on whether telemetry carries the relevant context, not simply on how much data a vendor stores.
The terms should not become a contest between old and new tools. A system can emit enormous amounts of data yet remain hard to understand. Another system can be modestly instrumented and easy to investigate because its events, service identities, and request relationships reflect how the application works.
The OpenTelemetry observability primer connects these ideas to understanding application behavior through telemetry. It provides a vocabulary, while each team still needs to decide what outcomes matter and what evidence will distinguish competing explanations.
Why the three pillars of observability need connections
Metrics summarize measurements over time. Request counts, duration histograms, and queue depth help detect changes and assess their scope. Logs record discrete events, ideally with consistent fields. Traces represent operations and their causal relationships across a request or workflow.
These are often called the three pillars of observability. The metaphor can mislead if it encourages three isolated storage systems. Their investigative value increases when a latency spike points to an example trace and that trace points to relevant logs. Consistent service names and deployment identifiers make those transitions possible.
Each signal also has limitations. A metric can hide an important minority inside an average. A log can omit what happened immediately before it. A trace can be incomplete because instrumentation or sampling excluded part of the path. An investigation should treat each as partial evidence.
Do not use one signal merely to imitate another. Logging every measurement as an unstructured sentence creates avoidable search and storage costs. Encoding individual customer identities as metric labels can produce excessive cardinality. Choose a representation that fits both the question and the operating constraints.
Distributed tracing explained through a checkout request
A trace contains spans. A span represents a unit of work with a start, an end, identifying context, and optional attributes, events, or links. A trace identifier groups related spans; a span identifier distinguishes one operation. Parent-child relationships often describe nested work, while links can represent other causal associations.
Imagine a gateway receives a checkout request and invokes an order service. The order service calls inventory and payment services. The inventory service consults a database. Instrumentation creates spans for these operations so their durations and relationships can be inspected together.
The OpenTelemetry traces documentation describes these building blocks. In practice, useful spans represent meaningful operations, rather than every function call. A span called “reserve inventory” can be more informative than dozens of generic method names.
Suppose the inventory span lasts 900 milliseconds, but its database child lasts only 30 milliseconds. The difference directs attention toward connection waiting, retries, application work, or missing instrumentation. It does not prove which explanation is correct. A trace narrows the investigation by showing where an account of elapsed time is incomplete.
Parallel calls require similar care. Adding all span durations can exceed the overall request duration because multiple operations overlap. Engineers should inspect the critical path and waiting relationships rather than equating a large sum of child durations with user-visible latency.
Context propagation makes the trace cross boundaries
Each service needs a way to continue the context received from its caller. For HTTP, the W3C Trace Context recommendation standardizes headers including traceparent and tracestate. Compatible instrumentation can extract incoming context and inject it into outgoing requests.
Propagation can break when a library, proxy, thread boundary, or message producer fails to carry the context. The result may look like disconnected traces even though the application performed one workflow. Test representative paths across the actual network and runtime stack.
Asynchronous work complicates the picture. An event can be consumed later, retried, or combined with other events. A parent-child hierarchy may not capture every relationship naturally. Span links can associate a processing operation with contributing contexts without pretending all work is one continuously nested call.
OpenTelemetry’s context propagation guide also discusses baggage, which carries additional application-defined values. Baggage should be used sparingly. Because it can travel downstream, putting secrets or personal data into it can spread sensitive information beyond its intended boundary.
An OpenTelemetry guide to the data path
OpenTelemetry provides APIs, SDKs, instrumentation, conventions, and collection components. It is an instrumentation and telemetry ecosystem, rather than a complete analysis backend. Teams generally export data to a backend that stores it and supports querying, visualization, and alerting.
Automatic instrumentation can quickly cover common HTTP frameworks, database clients, and messaging libraries. Manual instrumentation adds the domain-specific operations that libraries cannot infer, such as eligibility checks or reservation decisions. Start with automatic coverage, then fill the gaps that block real investigations.
The OpenTelemetry Collector can receive, process, and export telemetry. Depending on deployment requirements, it may run near applications or as shared infrastructure. It is useful for applying consistent routing and processing, but it becomes another production component to size and monitor.
Give every service stable resource attributes, including its name and deployment environment. Record version information so behavior can be compared across releases. Keep these identities consistent across signals. Otherwise, the team spends an incident reconciling naming differences instead of testing explanations.
A vendor-neutral collection path reduces some switching friction, but portability still requires discipline. Backend-specific queries, dashboards, and alert rules may need migration work. Export format compatibility is helpful without making all analysis environments interchangeable.
Sampling is a decision about evidence
Tracing every operation forever can be expensive and may impose unnecessary overhead. Sampling selects which traces to retain or export. The decision should reflect the questions the team needs to answer and the traffic patterns it actually operates.
Head sampling makes a decision near the start of a trace, before the full outcome is known. It is comparatively straightforward and can reduce downstream volume early. It can also miss a rare failure because the initial decision did not know that the request would become interesting.
Tail sampling evaluates completed or partially accumulated trace information before choosing retention. It can favor errors or unusually slow workflows, but requires buffering, routing, and operational capacity. Distributed traces must reach the relevant decision point coherently enough for the policy to work.
The OpenTelemetry sampling documentation explains these tradeoffs. A sensible policy might combine a representative baseline with extra retention for selected failures. The exact rates depend on volume, costs, and investigative needs.
Be explicit about selection bias. A backend containing mostly failed requests cannot directly answer what proportion of all requests failed unless the sampling arrangement supports that calculation. Keep authoritative traffic and error counters alongside sampled traces. “We found no example” is different from “the behavior did not occur.”
Structured logs should explain decisions
A structured log records fields that can be queried consistently: an event name, timestamp, severity, service identity, trace identifier, and a bounded set of domain details. The exact schema should help an engineer distinguish what happened and why.
For a rejected order, useful information might include a stable rejection code and the policy version applied. Recording the full customer message or payment payload is usually unnecessary for that purpose. Prefer the minimum evidence needed to diagnose behavior, with appropriate retention and access controls.
Avoid making log volume a proxy for completeness. Repeating the same stack trace at every layer can obscure the original failure. Capture the underlying error where it has the most context, then use correlation identifiers and meaningful state transitions to connect the story.
Log correlation also needs testing. A trace identifier printed into a field with inconsistent names will not automatically create a useful link. Verify that an engineer can move from an alert to a trace and from the trace to the relevant decision records in the deployed backend.
A worked investigation: the retry that amplified an outage
Consider an illustrative retailer whose successful checkout rate drops after a deployment. The first alert reports user impact. Regional metrics show the problem is concentrated in one cluster; deployment attributes show it began after a particular release. These observations constrain the investigation before anyone opens an individual trace.
A slow trace shows repeated inventory calls. Each attempt is individually short, but waiting and backoff consume the overall deadline. Correlated logs reveal that a new error mapping treats a permanent validation failure as retryable. Increased request volume then burdens the inventory service and affects otherwise valid orders.
The immediate action might be a rollback or a narrow configuration change, depending on the team’s tested recovery paths. The trace does not prescribe that action by itself. Engineers combine it with release knowledge, business impact, and evidence about whether the proposed intervention can safely reduce load.
The follow-up should examine the feedback loop. A local retry policy increased shared demand, which increased failure, which triggered more retries. The site’s article on systems thinking in software engineering and DevOps provides a broader way to reason about these interactions.
Add a regression scenario that checks the retry classification and deadline behavior. Improve instrumentation around retry count and remaining time budget. The useful outcome is a more understandable and controllable system, not simply a new dashboard displaying the incident’s symptom.
Instrument for questions, then verify the instrumentation
Begin with a small set of important user journeys. List the services, queues, and data stores involved, then identify where waiting or failure could become invisible. Add instrumentation around those boundaries and around consequential business decisions.
Next, exercise known scenarios in a controlled environment: a slow dependency, a rejected request, a retry, and delayed background work. Confirm that telemetry represents each scenario correctly. A trace that labels every request successful despite application-level rejection gives investigators the wrong picture.
Measure the instrumentation’s own overhead and failure behavior. Applications should have deliberate policies for full buffers, exporter outages, and collector backpressure. Telemetry is intended to explain production problems; its delivery path must not silently become a major source of them.
Finally, connect system understanding to an explicit model of the domain. The discussion of digital twins and system models offers related context: a model is useful when it helps explain reality, while its assumptions and missing observations remain visible.
Set a telemetry budget before volume becomes the problem
For the checkout service, divide telemetry needs into continuous measurement and retained examples. Continuous metrics establish the rate of successful purchases and the distribution of latency. Retained traces explain selected journeys. Logs explain decisions that would otherwise be ambiguous. These roles let the team reduce redundant data without losing the evidence it depends on.
Review cardinality at the point where an attribute is introduced. A deployment region has a bounded set of values; a complete request URL with arbitrary query parameters may not. Preserve a route template in metrics, and put carefully controlled request detail into a more appropriate signal when it is needed. This makes queries more predictable and reduces accidental disclosure.
Retention should also follow the question. Incident investigation, release comparison, and long-term capacity planning operate over different time horizons. Keeping every raw event indefinitely is rarely the only way to preserve useful historical evidence. Aggregation and selective retention can serve different purposes, provided their limits are documented.
Assign ownership to the telemetry schema and collection path. A service rename, changed attribute, or exporter update can break an investigation even while the application remains healthy. Include representative telemetry checks in release work for these changes. The team should know whether a missing span reflects missing application work or a change in what it observes.
Review a recent incident as a practical audit: could an engineer identify user impact, isolate the affected path, explain the decision, and verify recovery with the retained data? The unanswered question defines the next instrumentation improvement more clearly than a general request to collect more telemetry.
Review alert design with the same discipline. A useful page should identify the affected outcome, provide a starting investigation path, and point to an owner who can act. An alert that repeatedly fires without a decision becomes background noise. Use service objectives and observed failure patterns to decide what deserves interruption, then verify that the linked telemetry still works after releases.
What a useful observability program produces
The result should be shorter paths from user impact to a testable explanation. Monitor known service objectives, preserve context across boundaries, and combine traces with metrics and decision-oriented logs. Review the evidence gaps exposed by real incidents.
Distributed tracing matters because modern behavior is distributed too. No individual service owns the entire story. Observability improves when the team can reconstruct that story accurately enough to act, learn, and verify that its intervention worked.
