Every team that runs software in production eventually lands on the same three nouns. Metrics, logs, traces. Engineers talk about them like they are pillars of a temple, each one sacred, each one required. They are not pillars. They are three different answers to three different questions, and the skill that separates a calm on-call rotation from a panicked one is knowing which question to ask first.
Dashboards, log indexes, and tracing pipelines all feel similar from the outside. A new SRE (site reliability engineer, the person who gets paged when the system breaks) staring at three different graphs during an incident often reaches for the wrong one. What follows is when each signal helps, when it gets in the way, and how the three fit together.
Why three signals exist at all
Inside any running system, things happen. A user clicks a button. A database query takes 800 milliseconds. A pod crashes at 2:14 a.m. Each is an event, and the team needs some way to see what happened after the fact.
The industry settled on three recording styles for those events. Counters and gauges that capture trends over time. Free-form records that capture one moment in full. And stitched-together timelines that follow a single user request across every service it touched. None is older or more fundamental than the others. Each answers a different question, and a team that treats them as interchangeable ends up paying for storage they never read.
The first question to settle is which question you are trying to answer during an incident. Most newcomers skip this and reach for logs first because logs feel like a transcript. Usually the wrong move.
What metrics actually buy you
A metric is a single number, sampled at a regular interval, stored for ninety days or forever. The format is the same regardless of which platform you use. A name, a timestamp, a value, and a few labels.
Two metrics the same team might keep:
- http_requests_total{service=”checkout”, status=”500″} = 47 in the last minute
- database_query_duration_seconds{service=”orders”, quantile=”0.95″} = 0.83
Both answers are cheap to compute and cheap to store. A modern time-series database can hold a million distinct metric series across months of data. That is the first reason metrics lead most observability stacks.
The second reason is the alerting story. An alert that says “error rate above 5 percent for 5 minutes” is two lines of PromQL (the query language used by Prometheus, the most common open source metrics platform) plus a notification hook. An alert built on logs requires shipping log lines through a query engine in real time. An alert built on traces requires the trace to finish before you can check anything. Metrics are the only signal that tells you something is wrong while the thing is still wrong.
What metrics do not give you is the second half of any investigation. Once the alert fires, you know what is broken. You do not yet know where, and you do not know why. Metrics answer “is something off?” and answer it well. Everything else comes from a different signal.
What logs are really for
A log entry is a structured (or semi-structured) line of text, written by the application at a moment in time, containing whatever the developer thought would be useful later. A log is a confession the code makes about what it was doing.
A real log line from a web server:
2026-09-12T14:03:21Z app=checkout level=error msg="payment gateway timeout" request_id=8a7b-4412 upstream_status=502 retry=2
The fields are arbitrary. Each team picks the labels they care about. The shape is whatever you can grep for at 3 a.m. with one eye open.
Logs are the only signal that captures the reason something failed. When a metric tells you the error rate is climbing, the trace narrows it down to a service, and the log line tells you the upstream payment gateway timed out. That last hop from “I know what service” to “I know what to fix” almost always happens inside a log file. It is the only place where developer context ends up. The team that emits no useful log lines is the team that gets stuck at the second hop.
Storage is the catch. A busy service can throw off several gigabytes of log data per day, most of which is uninteresting startup chatter and successful request noise. Most teams learn this the hard way when the first monthly bill arrives and finance asks pointed questions. The response is a retention policy: keep 14 days at full fidelity, 90 days sampled, and throw away the rest.
What traces actually show
A trace is the assembled story of one request across every service it touched. Each piece of the journey (a database call, a remote API call, a cache lookup) is recorded as a span. The spans get stitched together using a shared trace ID that the first service generates and passes down through every downstream call.
A trace answers “where in this distributed mess did the time go.” Take a single checkout click that touches a frontend, an order service, a payment service, an inventory service, and three databases (eight stops in total). The metric says the click was slow. The log line at the front door says it was slow. Neither says which stop was the slow one. Only the trace breaks the total into its parts:
- Edge cache: 4 ms
- Frontend server: 18 ms
- Auth service: 11 ms
- Inventory service: 1,180 ms (database blocked on lock)
The slow stop jumps off the page. That is the value of a trace. The cost is that producing traces for every request is expensive. A typical trace is hundreds of spans. Most teams sample, saving only a fraction (1 percent, 10 percent, or only the failing ones). The trade is real: storage cost against visibility into the long tail.
Traces also have a dependency metrics and logs do not. Every service has to be instrumented (meaning every service has to explicitly opt into emitting spans when it makes calls) for the trace to be whole. If one service skipped that step, the trace breaks at the boundary and the time spent there is invisible. Most teams that adopt tracing spend the first quarter getting every service instrumented, and the slow ones show up in the gap.
Putting the three together during a real incident
A realistic runbook for a mid-size team during a production incident:
The alert fires. The metric for checkout error rate has climbed from 0.3 percent to 5.8 percent in the last seven minutes. Phone goes off. Engineer opens the dashboard. The metric tells them something is wrong and roughly how bad. It does not tell them where.
Engineer opens the trace view. Filters to failing requests from the same window. Most of them are stuck in the payment service. Specifically, a span called verify_fraud_score is now averaging thirty-four seconds instead of 180 milliseconds. The trace tells them where in the system the time went.
Engineer jumps into the log search for that service. The same line is repeating: timeout against an external fraud-check vendor’s API. The fraud vendor is having an outage. Nothing the team built is broken. They just have to roll over to the backup provider. The log tells them why.
Three signals, three questions, eight minutes total from page to fix. Pull any one out and the chain breaks. Metric alone tells you something is wrong but not where. Trace alone tells you where but not why. Log alone has the why but it is hiding among a million unrelated lines.
The thread that holds the three together is the trace ID stamped onto every log line. Without it, the engineer has to search by approximate time and hope. With it, one click on a slow span pulls up exactly the log entries it produced.
Trade-offs
Metrics are not free in everything except storage. The real cost is the cardinality (the number of unique label combinations) of the labels you choose. A metric with labels like service, endpoint, status_code, user_id produces millions of distinct time series, and time-series databases do not love that. Keep high-cardinality labels (user IDs, request IDs, raw URLs) out of metrics and into logs and traces.
Logs are not free in any meaningful sense. Storage grows linearly with traffic and the cost compounds. Retention policies are not optional, and the right retention depends on the team. The team that does not decide a retention policy will eventually have one decided for them by the bill.
Traces are not free in setup time. Every service has to be instrumented, and the slow ones (the legacy monolith nobody wants to touch) end up as gaps. Sampling is the hard decision. Most teams settle on head-based sampling (a random 1 to 10 percent of all requests get traced, decided when the request enters the system) plus tail-based sampling on errors (all failing requests get traced, decided when the request exits). The second part matters more than the first.
Vendor lock-in shows up in traces faster than metrics. Most tracing platforms use a proprietary wire format, and switching tools means re-instrumenting every service. OpenTelemetry (an open source project under the CNCF, the Cloud Native Computing Foundation) is the standard for emitting traces, but the backends still differ. If you are choosing a tracing backend, pick one you can leave. The lock-in cost is real and shows up in year two.
If you have one service on one server, you do not need traces and barely need metrics. Logs alone will tell you most of what is happening. If you have ten services and a request crosses seven of them, you need traces. There is no clever way to debug a distributed system without distributed debugging.
What I would tell past me
Three things for the engineer about to set up observability on a system that does not have any:
- Lead with metrics, not logs. A handful of dashboards covering the standard signals (latency, traffic, errors, saturation) catches most of what an on-call team needs to know in the first six months. The storage cost is near zero and the alert story is the simplest of the three.
- Make logs structured on day one. JSON logs cost about the same as free-text logs to store, and the difference between searching structured fields and grepping raw text is the difference between a five-minute investigation and an hour-long one.
- Stamp the trace ID onto every log line. That single integration is the highest-payoff move in the whole stack. Without it, you have three separate data sources. With it, you have one investigative workflow that flows between all three.