Most enterprises that ship AI features are running something they cannot fully see. A retrieval pipeline answers confidently while pulling the wrong context. An agentic workflow burns ninety seconds of compute before timing out. A fine-tuned model quietly degrades after a prompt template is tweaked. None of these incidents trigger a 500 error or surface in a standard APM (application performance monitoring) dashboard. By the time a user notices, the underlying behavior has usually been drifting for days.
This is the new operational gap. The fix has arrived faster than most teams realize. A wave of open source tooling, built specifically for the probabilistic, token-metered, prompt-driven nature of LLM applications, is finally production-ready. The architecture to instrument, monitor, and govern production AI is available, and it does not require buying a vendor bundle. The question is whether your team has built it yet.
What makes AI a different observability problem
Monitoring software used to mean watching for crashes, slow queries, and dropped packets. Those failure modes are deterministic: the code either throws or it does not, and the latency you measure today is the latency you will see tomorrow. LLM-driven systems break that contract in ways that traditional tooling cannot catch.
Consider the failure modes you actually have to track:
- The same prompt can return different answers on different runs, so a binary health check is meaningless
- A model can return a confident, well-formed response that is factually wrong, with no exception raised anywhere in the stack
- A pipeline might run fine at first and degrade as the prompt template drifts across deploys
- A workflow chain that runs across thirty calls might slow down because of any one node, and you cannot tell which node from a single 500
- An LLM call that succeeds but quietly burns ten times the tokens you expected becomes a bill problem long before it becomes a behavior problem
The honest framing is that AI observability is a different discipline. It needs new kinds of signals, not just more of the old ones.
Why open source is the right starting point
Vendor-native observability for AI is improving, but it tends to lock you in, hides its pricing, and limits how far you can extend it. When your LLM observability stack lives inside a model provider’s own console, you end up with visibility only for that provider. Open source changes the trade in three useful ways:
- It spans providers. A single instrumentation layer can watch Anthropic, OpenAI, Mistral, a local Ollama instance, and a Hugging Face endpoint, because OpenLLMetry and OpenTelemetry (OTel) sit underneath all of them
- It is extensible. Custom retrieval steps, proprietary vector stores, and homegrown orchestration all sit inside your codebase, and open source tools let you add custom spans to each one
- It caps the cost. Enterprise observability suites charge per seat, per event, or per gigabyte, and at LLM token volumes those fees can rival the inference bill itself. Open source shifts the spend to compute you already own
The most mature AI observability programs at well-run enterprises run on open source at their core. OTel for the data plane, Grafana for visualization, and a small set of LLM-native tools for the semantic layer.
The four-tier reference architecture
A production-grade AI observability stack can be modelled as four tiers, each with a clear job. The tiers do not need to be deployed in order, but if you skip any one of them you will hit a wall within a quarter.
| Tier | Role | Real-world building blocks |
|---|---|---|
| Instrumentation | Capture spans at every LLM, retrieval, and tool call | OpenTelemetry, OpenLLMetry, Helicone |
| Collection and transport | Aggregate, sample, enrich, and forward telemetry | OTel Collector |
| Storage and analysis | Persist traces, metrics, logs, and LLM semantic data | Tempo, Prometheus, Loki, LangFuse |
| Visualization and feedback | Dashboards, alerts, evaluation loops | Grafana, Alertmanager, Arize Phoenix, Evidently AI |
- Tier 1 instruments every AI component at the source. That means LLM API calls, vector store queries, retrieval steps, tool invocations, and orchestration decisions
- Tier 2 aggregates the stream and enriches it with environment metadata, so every span carries the model version, prompt version, deployment region, and cost center
- Tier 3 keeps infrastructure data and LLM semantic data on separate backends, because their retention, query patterns, and governance rules diverge sharply
- Tier 4 closes the loop. Failing traces get scored, the scores attach to the trace, and the result lands back in the prompt engineering or fine tuning queue that produced the regression
Treating observability as an architectural commitment, not a monitor bolted on later, is the difference between an AI product you can debug and one you can only hope is working.
The cloud-native layer underneath
OpenTelemetry is the gravitational centre. It is the CNCF (Cloud Native Computing Foundation) standard for telemetry, and it ships a vendor-neutral SDK that captures traces, metrics, and logs in any language your team already uses. For AI observability, OTel is the transport layer: every LLM-native tool either exports to OTel or is built directly on top of it.
Prometheus handles metrics storage and alerting. Grafana provides the dashboard surface. Loki aggregates logs cheaply. Jaeger and Tempo handle distributed tracing. None of this is novel, and that is the point. The work your team has already done to instrument microservices composes naturally with the AI-native layer that sits on top.
The LLM-native semantic layer
The semantic layer is where AI-specific tools live. LangFuse stands out as the open source platform for LLM observability, with prompt versioning, session-level tracing, token cost attribution, eval score capture, and a replay interface for debugging production failures. OpenLLMetry bridges the two worlds by auto-instrumenting LangChain, OpenAI, Anthropic, and other SDKs and emitting OTel-compatible spans, which means your LLM traces flow into the Jaeger or Tempo backend you already operate.
Arize Phoenix, Evidently AI, and MLflow handle the evaluation half of the loop, which is where observability stops being a passive monitor. Failing traces need to be scoreable, the score needs to attach to the trace, and the result needs to land back in the engineering queue that can fix the regression. Without that loop, observability is just an ever-growing record of incidents you cannot prevent.
Practices that hold up at enterprise scale
Getting the toolchain in place is the easy part. Operating it well takes discipline:
- Instrument at the pipeline level, not just the model. Capture the full span tree: retrieval, pre/post-processing, routing, and every tool call
- Set baseline SLOs (service level objectives) before production. Define p95 latency, token throughput, and hallucination thresholds before go-live, not after the first incident
- Separate observability by persona. Ops teams need infrastructure metrics, product teams need behavioral metrics, finance teams need cost attribution, and security needs audit logs
- Capture prompts and outputs selectively. Log full prompt-response pairs for debugging with sampling and PII scrubbing. Full capture at scale is expensive and risky
- Treat cost as a first-class metric. Token consumption maps directly to spend, so alert on token spikes and unexpected routing to a more expensive model
- Use semantic versioning for prompts. Treat prompts as deployable artifacts and correlate observability data with prompt versions so regressions are easy to isolate
- Plan for multi-model and multi-provider visibility. Abstract instrumentation at the gateway layer so you get unified visibility across providers without re-instrumenting every service
Trade-offs worth naming before adoption
Open source AI observability is not free, even with no license fees. The first cost is integration work. Every team adopting OTel, LangFuse, or OpenLLMetry has to invest in instrumentation, schema design, and dashboarding before they see returns. The second cost is governance. Open source tools rarely ship the role-based access control, audit logging, and retention policies that regulated industries require, so you have to layer your own controls on top. The third cost is evaluation. Observability tells you what happened; evaluation tells you whether what happened was good. Skip the evaluation feedback loop and you get visibility without the ability to act on it.
A useful mental model is that observability is a prerequisite for operating AI responsibly, not a substitute for it. Teams that treat it as an architectural commitment, with clear owners, a documented tier structure, and a tight loop back into prompt engineering, recover from regressions in days. Teams that treat it as a monitoring bolt-on spend weeks chasing issues across disconnected dashboards and never get ahead of the next release.
A short adoption path
If you are starting from scratch, the cleanest first move is to deploy OpenTelemetry across the services you already run, then add OpenLLMetry to instrument the AI components. Wire those traces into a Tempo or Jaeger backend, point Grafana at it, and you have the foundational layer in place. From there, layer in LangFuse for prompt versioning and semantic session data, then bring in Arize Phoenix or Evidently AI for evaluation. The temptation to skip the foundational layer and jump straight to LLM-native tools is real, but resist it. The semantic layer is only useful when the transport underneath it actually delivers telemetry to it.