If you ship an LLM feature in production and your telemetry looks the same as it did for the REST API you ran before it, you are going to miss the bugs that actually matter. LLM applications (software built on top of large language models such as Claude, GPT, or Gemini) do not crash the way classic services do. They finish requests with HTTP 200, return grammatically clean prose, and quietly hand the caller something wrong. A green health check does not mean the system is healthy. It means the process is up.
The shift from deterministic to probabilistic systems changes what “instrumented” means. You are no longer measuring whether the code path executed. You are measuring whether the output was correct, safe, and worth the tokens it spent. That is a different problem, and it needs a different telemetry stack.
What classical observability misses
Traditional observability sits on three pillars. Metrics tell you request rate, error rate, and latency. Logs tell you what each component said. Traces (records of a single request’s path across services, stitched together by an ID) tell you where time was spent. The MELT model (Metrics, Events, Logs, and Traces, a common shorthand in the APM world) is enough for a service that, given input X, always produces output Y.
LLM services break that assumption. Same prompt, different Tuesday, different answer. The classic “is it up?” probe returns 200 even when the answer is fabricated, biased, or a security risk. A P95 latency dashboard tells you nothing about whether the answer was hallucinated (the model produced a confident-sounding claim with no basis in the input or training data). You need a parallel scoreboard for output quality, grounded against whatever authoritative source your application is supposed to be drawing from, and that scoreboard is what the AI observability layer is for.
I have watched this play out on three internal tools in the last year. In each case, the team’s first detection of a regression was a customer email, not a graph. The graphs all showed green. We had built an excellent dashboard for “the system is on fire” and zero visibility into “the system is confidently wrong.”
The deeper issue is that classical observability was built for systems whose failure modes are observable. An HTTP 500 is a failure mode. A four-sentence answer that contradicts the source document is a failure mode nobody has wired up. You cannot alert on it, you cannot page on it, and you cannot even detect it without ground truth.
The four pillars worth instrumenting
The shape that has held up across the tools I have looked at breaks AI observability into four layers, not three. The layers map to what you actually need to investigate when something is off, and skipping any of them leaves a gap that bites you the next time production drifts.
Performance is the layer that overlaps most with classic observability. Time to first token (TTFT, the delay before the model starts streaming output), end-to-end latency, and queue depth tell you whether the pipeline is degraded. The interesting failure mode here is a slow upstream, things like a vector database under load, an embedding service that is throttled, or a rate limit you did not know you had. Classic APM handles most of this layer well; the AI-specific touch is correlating latency to the prompt template version that produced each request.
Output quality is where MELT alone falls down. You need to measure whether the answer was right, and that means scoring the output. The most common scoring systems use an LLM as a judge against a labelled rubric (a small set of correct reference answers written down ahead of time) for hallucination, groundedness (whether the output is actually supported by the retrieved context), toxicity, and PII leakage (Personally Identifiable Information, anything that could identify a real person). Numbers from a real production traffic sample beat subjective review from your team every time.
RAG and context diagnostics come next if your application does retrieval-augmented generation (RAG, the pattern where the app fetches relevant documents from a vector store and hands them to the model as context). When an answer is wrong, the first question is usually not “what did the model do” but “what context did we feed it.” You want to log the chunk retrieval scores, the prompt template version, and the embeddings used. A drift in chunk relevance is almost always upstream of a drift in answer quality, and discovering that relationship without a log is pure luck.
Cost and infrastructure form the fourth pillar. Token spend per user, GPU and CPU utilization against request volume, and per-tenant accounting belong here. Cost is the easiest layer to skip until the bill arrives, and the hardest layer to bolt on after the fact.
A short list of signals that pay for themselves first
These are the metrics I would instrument on day one, ranked by the ratio of signal to setup cost.
- Hallucination rate sampled against a labelled rubric of 200 real prompts. Run this sampling hourly, not on every request. Costs almost nothing, catches the common regressions.
- Cost per resolved task, not per request. Two requests can have wildly different token costs depending on tool calls and retries. The unit of value is the task that actually got done.
- TTFT p95 by prompt template. A specific template slowing down is almost always a sign that an upstream dependency is struggling, and you want to know which template.
- Groundedness score for RAG responses, scored by an LLM-as-judge. Calibrate against human review on 50 examples before you trust the number.
Tools I would actually evaluate
The vendor landscape is large enough that “which tool” is now downstream of “which stack are you already running.”
If your team is already on Datadog, the LLM Observability module is the path of least resistance. The win is that low-level infrastructure metrics, prompt latency, and error data land on the same dashboard. The cost is the price floor Datadog charges for it, which is not trivial for small teams.
Langfuse is the strongest open-source option I have used end to end. It is MIT-licensed, has solid SDK hooks for the major frameworks, and exposes prompt history, user behavior, and scoring side by side. For a small team that wants to own its own data without sending traces to a SaaS vendor, it is the natural default.
Arize Phoenix has the strongest local debugging story and the cleanest OpenTelemetry compliance (OpenTelemetry, or OTel, is an open standard for emitting telemetry data; compliance means the tool plays well with collectors you already operate), which matters when you want to route through your existing collectors. Its RAG-specific evaluation tools are the best of the group if retrieval quality is your dominant concern.
Helicone is the lightweight proxy option: drop it in front of any provider, swap a base URL, and you get edge caching and pricing telemetry in minutes. It is not a full observability platform, but it is the fastest way to learn whether your costs are bounded before you commit to something heavier.
Trade-offs
None of these tools are free, even the open-source ones. Langfuse demands a Postgres database, a working deployment story, and at least one engineer willing to own the upgrade cadence. Phoenix is friendlier in dev mode but heavier to operate in production, and its local-only debugging model does not survive the first person who needs to investigate from a phone. The SaaS options trade data residency (where your telemetry physically lives, often a compliance question for healthcare or finance workloads) for less operational burden. The proxy option (Helicone) gives you pricing and caching but nothing about quality, so you still need a separate scoring story. A real setup ends up pulling from two or three of these, not one.
The timing question matters too. Adding AI observability before shipping costs you a few days of setup and protects you from a six-month investigation the first time something quietly breaks. Adding it after shipping costs you your first incident to figure out the shape of the problem, which is the exact thing AI observability exists to prevent.
Bottom line: Treat LLM output quality as a first-class telemetry dimension, not a Notion page someone fills out by hand. Pick the tool that fits your existing stack first. If you do not have a stack yet, Langfuse is the open-source default; if you already pay Datadog, turn on LLM Observability tomorrow morning. Either way, ship the scoring layer in the same sprint as the first prompt template, not the one after.