Most AI applications ship without real observability. They have logs, request counts, maybe a Grafana dashboard with token usage over time. What they lack is the layer that lets an engineer answer a specific question after a production incident. Why was this answer inaccurate. Which retrieval step introduced irrelevant context. Whether a prompt template regressed after a recent change. Why latency spiked for one model and not another. How a five-line prompt change doubled token consumption overnight. Traditional application monitoring was designed for software where the same input reliably produces the same output. An AI system does not behave that way. The cost dimension is also different: LLM applications are not just compute-intensive, they are usage-sensitive.
This is why observability for AI has become its own discipline. The questions are different from web application monitoring, the signals are broader, and the cost dimension is central. The good news is that the open source stack for it has matured faster than most teams have noticed.
The shift from application monitoring
The textbook definition still applies: observability is the ability to infer internal system state by examining external outputs. For AI systems the operational questions are broader. Teams are no longer asking only whether a request failed. They want to know whether a prompt template regressed after a recent change, why latency increased for one model, and how token costs spiked after a rollout.
A serious observability approach combines traces, metrics, logs, and evaluations, each layer serving a different debugging question. Traces record how a request moves through retrieval, prompt assembly, model invocation, tool usage, guardrail checks, and the final response. Metrics roll up the latency, throughput, token usage, cost, failure rates, and response patterns into a single dashboard. Logs retain the detailed execution events for post-incident forensics. Evaluations sit alongside the operational layer and answer a quality question: is the response grounded, relevant, safe, and on task.
The financial dimension matters as much as the engineering dimension, because cost moves with usage rather than with infrastructure. Lengthening a prompt, deepening the retrieval step, switching models, or extending conversation memory can each change how many tokens a request consumes and therefore how much the bill comes to at the end of the month. End-to-end tracing makes those shifts visible. It also provides the audit trail that compliance teams need: which model handled the request, what context was passed in, whether sensitive data leaked into prompts or outputs, and how long each step took. For teams operating under security, privacy, or compliance constraints, that audit trail is not optional.
What the open source stack looks like in 2026
Open source tooling for AI observability has evolved quickly. The strongest implementations combine more than one component: a standard telemetry layer, a trace collection pipeline, and a specialised interface for AI-specific analysis. Choosing among them depends on architecture, privacy requirements, engineering maturity, and whether the team prefers SDK-based integration, proxy-based capture, or a hybrid model.
The interesting question for 2026 is not whether to adopt AI observability but which layer to start with. The choice is shaped by what the team is trying to learn. A team that needs deep visibility into model behaviour and wants to integrate with the rest of an existing observability stack will reach for a standards-based approach first. A team that needs quick answers about why a chatbot is giving wrong answers will reach for an evaluation-focused tool first. A team whose immediate problem is cost will reach for a proxy first.
The standards-based path
OpenTelemetry is the underlying telemetry layer; OpenLLMetry sits on top and instruments the model calls the standard does not cover by default. Together they capture invocations, prompt and completion metadata, token usage, retries, errors, and the way an application talks to frameworks like LangChain and LlamaIndex. The reason this pair has become the de facto starting point is portability: traces produced by OpenLLMetry can be shipped to whatever backend the team already runs, whether that is Jaeger, Tempo, Datadog, or Honeycomb.
The evaluation-focused path
For teams whose first question is “is the answer actually correct,” Phoenix and TruLens are the most common starting points. Phoenix gives a visual interface for inspecting RAG pipelines, comparing outputs, and tracking quality across prompt revisions. TruLens focuses specifically on whether the response was grounded and relevant, which makes it well suited to environments where trust and accuracy matter more than raw throughput. Langfuse is the third option here, and it is the most common choice for teams that want one platform that handles tracing, prompt observation, version tracking, and evaluation in the same interface.
The proxy path
Helicone works as a proxy that sits in front of the model API rather than inside the application. Every request flows through it, and the proxy records usage, latency, cost, and response metadata without requiring code changes. The trade-off is that it sees the request and response but not the inside of the model call, so it is a fast starting point for cost and traffic visibility but not for deep instrumentation. OpenLIT captures similar signals through instrumentation rather than as a proxy, which is the better fit for teams that want self-hosted capture.
Editorial take: pick a layer, not a tool
The pattern that works in practice is to pick one layer and stay there until it is producing useful answers. Mixing standards-based instrumentation, evaluation platforms, and proxy capture from day one produces a stack that is hard to reason about and produces contradictory signals when something goes wrong. Pick the layer that matches the most pressing question. Move to a second layer once the first one is consistently producing useful answers.
The mistake teams make most often is to deploy everything at once. With a tool-rich category, that produces more telemetry than anyone can read, and the actual signals that would catch a regression get buried under the noise. A team that starts with OpenTelemetry and OpenLLMetry will have a portable foundation that integrates with the rest of their infrastructure. A team that starts with Arize Phoenix or TruLens will have an answer to the quality question quickly. A team that starts with Helicone will have a cost and traffic picture within hours. None of these is the wrong choice if it matches the actual question.
Where to start
A reasonable default for most teams is to start with OpenTelemetry as the standard telemetry layer, instrument one or two critical LLM call sites with OpenLLMetry, and let traces flow into a backend the team is already comfortable with. Add an evaluation tool such as TruLens or Arize Phoenix when quality regressions become a recurring problem rather than a one-off. Reach for a proxy like Helicone only when the goal is fast visibility with minimal code change rather than deep instrumentation.
The key decision is whether the team is debugging an immediate quality problem, building a long-term observability practice, or trying to keep an eye on cost. The first favours evaluation-focused tools. The second favours standards-based instrumentation. The third favours proxy-based cost capture. Mixing all three is fine, but only after the first deployment is producing useful traces.
Trade-offs
- Standards-based instrumentation gives the cleanest long-term story but requires upfront work to wire into every model call. Proxy-based capture is faster to deploy but does not see inside the model invocation.
- Evaluation tools add real value when groundedness or relevance regressions are recurring. For systems where output quality is already tightly controlled, they are overhead rather than insight.
- The open source stack is still consolidating. Picking a tool today carries some risk that it is not where the field will be in eighteen months. OpenTelemetry-based instrumentation is the safer bet for portability.
- Token cost and latency correlate with prompt length and retrieval depth, but not linearly. End-to-end tracing is the only way to see where cost actually accumulates.
Bottom line
AI observability is a different discipline from conventional application monitoring. The questions are different, the signals are broader, and the cost dimension is central. The open source stack is mature enough that a serious team can build a full practice without paying for a SaaS product, but the layer to start with depends on whether the immediate problem is quality, latency, or cost. Standards-based instrumentation through OpenTelemetry and OpenLLMetry gives the cleanest foundation; evaluation tools add depth once the basics are in place; proxies offer fast visibility when deep instrumentation is not yet feasible. Most teams will end up using two or three of these in combination, and the order in which they adopt them matters more than the final combination.