WierdX — Programming Reference All tutorials →
Developer reference · Practical tutorials · CS fundamentals
Systems & Infrastructure

Observability: Logs, Metrics, and Traces Explained

A dashboard full of green metrics can coexist with a broken feature nobody's noticed yet. Logs, metrics, and traces answer different questions, and a system missing any one of them has a permanent blind spot.

Published July 6, 2026

"Observability" gets used loosely, but it rests on three distinct kinds of data, each answering a question the other two can't answer well. Metrics tell you that something is wrong. Logs tell you what happened in a specific case. Traces tell you where, across a chain of services, the time or the failure actually occurred. Skipping any one of the three means falling back on guesswork exactly when you can least afford to guess.

Metrics: aggregate numbers over time

A metric is a number tracked over time — request count, error rate, p99 latency, queue depth, CPU usage. Metrics are cheap to store because they're pre-aggregated: instead of keeping every individual request, you keep counters and histograms that summarize thousands of requests into a handful of numbers per time bucket. This is what makes metrics good for alerting and dashboards — "p99 latency exceeded 2 seconds for 5 minutes" is a clean, cheap condition to evaluate continuously.

What metrics can't tell you is which specific request was slow, what its parameters were, or why. A latency histogram shows you the shape of a problem without a single example of it.

Logs: what happened, in detail, one event at a time

A log line records a specific event with whatever context you chose to include. Structured logging — emitting logs as JSON objects with consistent field names rather than free-form text — is what makes logs searchable at scale:

{
  "timestamp": "2026-07-06T14:22:03Z",
  "level": "error",
  "message": "payment charge failed",
  "user_id": "u_8842",
  "order_id": "ord_5591",
  "provider_error": "card_declined",
  "request_id": "req_a91f3c"
}

Structured fields let you filter — "show me every failed charge for this user" — instead of grepping through unstructured text and hoping the message format matches your search. The request_id field above is what connects a log line back to a specific user-facing request, which matters once you're searching through millions of log lines from many service instances at once.

Traces: following one request across service boundaries

A single user action — "place an order" — might touch an API gateway, an orders service, an inventory service, a payment service, and a notification service. If the whole thing takes 800ms and feels slow, logs from five separate services tell you five separate stories that you have to manually stitch together by request ID and timestamp. A trace does that stitching automatically: it's a tree of timed spans, one per unit of work, all tagged with the same trace ID as the request propagates from service to service.

trace_id: 7f3a9c2e
├─ span: api-gateway (12ms)
├─ span: orders-service (760ms)
│  ├─ span: inventory-check (40ms)
│  └─ span: payment-service (690ms)
│     └─ span: card-provider-call (670ms)
└─ span: notification-service (8ms)

The trace above makes the answer obvious in a way scattered logs wouldn't: almost the entire 800ms is spent waiting on the external card provider inside the payment service, not in your own code. Without tracing, this kind of cross-service latency problem often gets misattributed to whichever service happens to log the slowest-looking line, when the actual time is being spent one or two hops further downstream.

OpenTelemetry has become the standard, vendor-neutral way to instrument code for traces (and metrics and logs) without locking your instrumentation to a single vendor's SDK; its documentation is a solid reference for how span propagation actually works across service boundaries: opentelemetry.io/docs.

Where each one falls short alone

Metrics alone tell you the p99 spiked but not which requests, users, or code paths were involved. Logs alone, without a shared request ID, are a pile of disconnected events with no way to reconstruct the path a single request took through your system. Traces alone are expensive to sample at 100% under high traffic, so most systems sample only a fraction of requests, which means the one specific failing request a user reports may never have been traced — you still need logs and metrics to catch and investigate the cases tracing didn't happen to sample.

Correlating all three

The practical payoff comes from linking them: a metric alert fires on elevated error rate, you pull up traces for requests in that time window tagged with errors, a trace shows which specific service and span is failing, and you jump to that service's logs filtered by the trace ID to see the actual error message and payload that caused it. Each layer narrows the search from "something, somewhere, is wrong" down to a specific line of a specific log for a specific request — the three data types are complementary evidence, not three equivalent ways of monitoring the same thing.