A leaderboard number says almost nothing about whether the model you just shipped is helping the people you shipped it to. That gap is not bad luck. It is the predictable result of measuring the wrong thing in the wrong place at the wrong time. If you treat the benchmark as the verdict, you will be right on paper and wrong on a Tuesday afternoon when a user with a corrupt CSV hits the feature you spent six weeks polishing.
The fix is not more benchmarks, and it is not a bigger model. The fix is a small set of habits that force your offline number and your online reality to argue with each other. This piece walks through those habits in the order you actually need them. I use the GitHub secret-scanning rollout as the running example because their situation is unusually clean: a public scoring system, a private definition of success, and a hard safety constraint. The lessons travel to any LLM feature where the wrong failure mode is more painful than the right one.
Name the failure you cannot afford, in one sentence
Most LLM evaluations start with a number. They should start with a sentence. The sentence is the wrong-kind-of-failure statement, written in plain English, naming the cost of being wrong in the most expensive direction. Until that sentence exists, every other choice in the eval pipeline is decoration.
The GitHub secret-scanning team wrote theirs first: “miss a real credential.” Everything that followed came from that. They knew the metric they could push on (precision, because noisy alerts were the developer-facing pain), and the metric they could not afford to break (recall, because a missed credential was the kind of error that does not get a second chance). They could defend any shipping decision by asking whether it moved precision while keeping recall inside an acceptable band.
You can copy the pattern without copying the specifics:
- Decide the user benefit in one sentence. Not “the model is good.” The specific thing the user gets when the system works.
- Decide the failure you cannot afford in one sentence. The cost of being wrong in the most expensive direction.
- Pick one metric to optimize for the benefit. Pick a second metric to constrain the failure. Do not combine them.
- Treat every other score in your dashboard as a tiebreaker, not a goal.
That is the only part of the framework that survives contact with reality. The rest is bookkeeping.
Mirror production in your eval set, or stop calling it an eval set
Here is the uncomfortable finding that pushes most teams back to the drawing board. The cleanest eval sets are the least informative. Curated benchmarks are useful for proving an idea is plausible in theory. They are terrible at predicting production behavior, because production traffic looks nothing like curated test data.
The GitHub team learned this when benchmark-grade strings turned out to be useless in real repositories. The shape of the noise was nothing like what the eval had been measuring. There were example keys planted in tutorials and never cleaned up. There were placeholder secrets in test fixtures that had been there for years. There were credentials that had been rotated months ago and were sitting in history git. None of those would ever be flagged in anger. All of them were being flagged. The benchmark said the model was right. The production data said the model was annoying a lot of people.
The fix is not subtle. Build a dataset that mirrors what the system will actually see, with all the noise that implies. A useful starting list:
- Pull real samples from production traffic, even a few hundred, and label them with the same standard your reviewers will use.
- Include obvious mistakes, edge cases, and at least one example of each known failure mode from your incident log.
- Keep the eval set versioned, so a regression on a fixed bug is visible rather than silently averaged out.
- Refresh the set every quarter, because user behavior drifts and the eval set has to drift with it.
- Mark the boring cases as boring. Do not delete them. Real traffic is mostly boring.
A clean benchmark that does not look like production is a story you tell yourself. A noisy dataset that does is a measurement.
Run both systems in parallel before you cut over
The most reliable way to ship an LLM feature that behaves the way you expect is to send the same live requests to both the old system and the candidate, side by side, for long enough that unusual days have a chance to show up. Anything else is a story you tell yourself with numbers you happen to like.
Shadow evaluation (sometimes called a dark launch) is not glamorous. It is also the only honest way to compare a candidate against the system you already have. Forward a copy of every production request to both systems. Score both on the same input so the comparison is fair. Run it for weeks, not days, since real traffic has weird edges and end-of-quarter spikes. Pipe the disagreements into your eval set rather than just your incident tracker.
If you cannot run the new system in parallel against live traffic, you are not ready to ship it. You are ready to ship a demo. Conflating the two is the second most common mistake after “we will average precision and recall.”
Rank your metrics instead of averaging them
Most teams throw precision and recall into a single F1 score and call it a day. That works when mistakes are equally painful. It hides everything when they are not. The GitHub team made the distinction explicit and it is the move I now recommend to anyone evaluating an LLM for production.
Hit rate, in plain terms, is the share of times the system is right when it chooses to act. A high hit rate means fewer wasted actions. For secret scanning, a high hit rate means developers stop getting paged for fake credentials. That is the user benefit. Coverage is the share of true cases the system actually catches. Low coverage means missed credentials. That is the safety risk.
Ranking the two numbers gives you a defendable shipping decision. Averaging them takes that defense away. Put the user-facing metric in the objective slot and let every experiment try to push it higher. Put the safety metric in the constraint slot and refuse to ship anything that drops it past your floor. If a candidate change cannot beat the objective without breaking the constraint, reject it.
This sounds obvious. Almost nobody does it, because a single average score is easy to put on a slide and the truth is rarely that symmetric.
The post-launch loop is where the work really starts
The phase most teams underestimate is everything that happens after the launch announcement. A model that performs well on day one will slowly drift out of shape, and the only way to notice is to keep measuring.
Treating LLM evaluation as a one-time gate is the second biggest mistake teams make, after trusting benchmarks. The loop has to outlive the launch announcement. Inputs evolve, the model behind the API changes, and the prompt you shipped six months ago stops fitting the traffic it actually sees. If you do not build a way to detect that drift, the first time you hear about it will be a customer escalation that names your company in a forum thread.
The shape of the loop is straightforward. Pick the production signal that proves the user benefit is still being delivered (precision, in GitHub’s case) and watch it. Pick the production signal that proves the constraint is still being met (recall) and watch it too. Wire alerts to both. Every time the system and a human reviewer disagree, that disagreement is data: it belongs in the next version of the eval set, not in a Slack thread that nobody reads again.
That is it. No magic. Just the discipline to keep watching after the launch party ends.
Trade-offs
The discipline above has a real cost. Building a production-shaped eval set means labelling samples by hand, which takes engineering time you would otherwise spend shipping. Shadow evaluation doubles the compute you are paying for during the rollout window. And the loop you put in place after launch is ongoing work, not a project with an end date.
You also have to live with the fact that your offline number will get worse the moment you switch from a clean benchmark to a noisy production-shaped set. That is a feature, not a bug. If your offline number does not get worse, your dataset probably still looks too clean to predict real behavior.
The honest upside is that the cost shows up early, in lab time and shadow traffic, instead of late, in incident reports and customer escalations. Most teams find that the early cost is the cheaper of the two.
Coach’s Note
If you take one thing from this piece, take this. Pull up a blank document and write, in plain language, the single sentence that explains what your LLM feature is supposed to do for the user, and the failure you are least willing to ship. Then check whether your current eval set can tell you, on its own, whether you are getting closer to that sentence or further from it. If the answer is no, you have just located the most expensive bug in your project, and the fix is on you, not on the model.
The move for this week is short. Sit down with the person who owns the product decision, write the sentence together, and rebuild your offline eval set around it. The leaderboard score will get worse. That is the work.