If you have ever paged someone at 2 a.m. because a checkout flow looked slow but every dashboard said “green,” you already know the cost of running infrastructure you cannot actually see. Vendors sell the fix as “add observability.” That is true, and it is also vague, because the word covers a lot of ground and a lot of vendor bills. Here is the question that actually matters: what does your system tell you when it is broken, and how fast does it tell you?
Observability is the practice of answering that question with data you already generate. Logs, metrics, and traces are the three signals most teams agree on, and the open source ecosystem has built real tooling around them. Prometheus (metrics), OpenTelemetry (the collector and SDK layer), Grafana, Loki, and Tempo are not a stack you must take as a bundle. They are a set of choices you can mix.
The reason this matters now, in 2026, is that the systems we are running are not the same as the ones the old monitoring tools were designed for. A single Linux box running a single Apache process is easy to monitor. A payment service that fans out across seven microservices, three managed databases, a queue, and a CDN, with a recommendation engine on the side, is not. When something goes wrong in that web, “the CPU is fine” tells you nothing. You need to know which hop is slow, which one is dropping requests, and which one is the one your customers actually feel.
What observability actually answers
Vendors pitch observability as a feature. It is more useful to think of it as a question you can ask: given the data my system is producing right now, can I explain why it is behaving this way? A simple system lets you answer with metrics. A complex one needs logs and traces too, because the failure is no longer in a single component.
Three signals each answer a different question:
- Metrics tell you how much, how fast, how often. Spikes in 95th percentile latency, drops in requests per second, sudden climbs in error rate. Metrics are cheap to collect and easy to alert on, which is why they are where most teams start.
- Logs tell you what happened, in narrative form. A log line is a timestamped statement about an event. Good logs include enough context to reconstruct what a request did, what it saw, and what it returned.
- Traces tell you where time went across a single request. A trace is a tree of spans, each one representing a unit of work, each one timed. When you have a trace, you can see that the slow request was not the database call, it was the third-party recommendation service that took 800 ms while everything else took 50 ms.
Metrics light up first. Logs explain what happened. Traces show you why.
The open source toolkit, and what each piece is for
Names get thrown around interchangeably, so it is worth pinning them down. Each tool has a specific job in the observability stack:
- Prometheus is the time-series database (a database optimized for storing values tagged with timestamps) most teams use for metrics. It pulls values from your services on a schedule, stores them, and lets you query them with its own query language. It is the de facto standard for metrics in a Kubernetes setup, and most other tools in this space know how to talk to it.
- OpenTelemetry is the collector. It is a project under the Cloud Native Computing Foundation that provides a single SDK and a single agent for getting logs, metrics, and traces out of your application and into a backend. You instrument once with OpenTelemetry, and you can send to Prometheus, to Jaeger, to a vendor, to a file, wherever you want.
- Grafana is the dashboard. It is vendor-neutral and connects to dozens of data sources, so you can put your Prometheus metrics, your Loki logs, and your Tempo traces on the same screen. When something is on fire, you want one place to look.
- Loki is a log store designed to be cheap and to work well with Grafana. Tempo is a trace store built for the same reason. You do not have to use both. You do have to know which signal you are looking for, because the answer you get depends on which one you ask.
Where teams get this wrong
Most common mistake is treating observability as a logging problem. Buy a log vendor, point it at your containers, and hope. Logs are necessary, but they are not enough. If you cannot measure the latency of a request across services, you cannot find the slow one. If you cannot correlate a log line with the trace that produced it, you cannot tell whether an error was caused by something inside your service or by something upstream.
Another mistake is treating every signal as free. Each metric you collect has a cost. Each span you emit has a cost. Each log line you keep for thirty days has a cost. The trick is to instrument the parts of your system where the answer matters, and to sample the parts where it does not. OpenTelemetry’s sampling layer is built for this. You can record every error and a fraction of the successful requests, which gives you most of the diagnostic value at a fraction of the cost.
A third failure mode is ignoring the trade between breadth and depth. A wide, shallow setup (metrics on everything, traces on a few things, logs on the rest) is easier to maintain and gives you most of the value. A deep, narrow setup (full traces on every request, full logs on every service) is more useful in a crisis but costs more to run and maintain. Most teams should start wide and shallow.
Trade-offs
Biggest trade-off is do-it-yourself versus a managed platform. The open source stack is free and flexible, but it is real work to run. You have a Prometheus, a Loki, a Tempo, a Grafana, and an OpenTelemetry collector to keep alive. A managed platform buys you time and reduces the number of things that can page you at 2 a.m., but it costs money and it locks you in to a vendor’s choices about retention, sampling, and query language.
Second trade-off is breadth versus depth. Most teams should start wide and shallow, and go deeper only after the wide setup is paying for itself.
Third trade-off is data retention. Metrics are cheap to keep; traces are not. A 30-day retention window for metrics and a 7-day window for traces is a reasonable default. Logs depend on what you need to debug, but a year of debug logs is a bill you probably do not want.
What I would tell past me
Start with metrics, because metrics are how you know whether anything is wrong. Add tracing on the path you actually care about, which is usually the request path your customers see. Add logs last, and only on the parts where you cannot get the answer from metrics or traces. Pick a backend, any backend, and start. The perfect stack that never ships is worse than the messy one that does.
If you are picking one tool to learn first, learn OpenTelemetry. It is the layer that lets you change your mind later, and it is the layer most teams underestimate. The collector is where you decide what gets sampled, what gets dropped, and what gets sent where. Get that right and the rest of the stack is interchangeable.
Observability is not a product you buy. It is a question you can answer, and the answer is only as good as the data you collect.
Source: https://www.opensourceforu.com/2026/08/observability-the-nervous-system-of-modern-digital-infrastructure/