There is a particular kind of engineering meeting I have learned to dread. Someone opens a slide. The slide has a line on it. The line is going down. The room nods. Someone says “great work” and moves on. Six weeks later we find out the actual product is slower, the user complaints are louder, and the metric on the slide kept going down the whole time. The slide was not lying. It was just measuring something irrelevant.
This is the 2026 disease of AI agent engineering. Per-call token counts, tool latency dashboards, and single-step cost graphs are easy to build and almost always misleading. They tell you about a slice of the workflow. They hide the fact that the slice is the wrong slice. GitHub’s Copilot team just published a postmortem of their own version of this mistake, and it is worth reading even if you never ship a coding agent, because the trap generalizes.
What happened to the Copilot team
Their story starts with a tool called RTK. RTK is a clever little utility that strips shell command output down before the model reads it. The premise was the obvious one. Less input means cheaper calls means faster responses. On a single-call benchmark, the numbers looked like a free lunch. The model, however, started reopening the original file or rerunning the command to recover details it had lost. The per-call number dropped. The full task cost climbed. The dashboard did not catch this, because the dashboard was not designed to catch this. Dashboards never are, when they only measure a slice.
That is the version of the disease I want to talk about. Not the tools themselves. The disease is the worship of the slice.
Why the slice is almost always the wrong thing to optimize
The reason slice metrics feel so good is that they have clean numbers. Every API call returns a token count. You graph it. You watch it go down. You feel productive. Meanwhile the user is waiting twice as long for an answer that is half as good, but no dashboard shows you that.
I have watched this exact pattern ship at three different companies now. The shape is always the same. A team finds a clever per-call optimization, ships it behind a feature flag, watches the dashboard improve, graduates it to a percentage of traffic, watches the dashboard improve more, ships it to everyone. Six months later somebody realizes the agent is making twice as many turns to converge. The dashboard kept going down the whole time. Nobody questioned it because the dashboard was the source of truth.
The pattern is the same every time. You cannot measure what you are not measuring. If your dashboard only sees the slice, the slice is what you will optimize, even when optimizing it harms the whole.
The one question that catches fake wins
When someone proposes a change that “reduces tokens per call,” I ask one question. Did the user get a better outcome, or did the user get a smaller bill for the same outcome. If the answer is “smaller bill for the same outcome,” the change probably does not matter. If the answer is “worse outcome,” the change optimized in the wrong direction. If the answer is “better outcome at higher per-call cost,” you have found something real and the dashboard needs an update.
This question is annoying because it cannot be answered from the dashboard. It requires looking at the actual product. It requires a human evaluating whether the agent finished the job. That is the work. The dashboard is supposed to support the work, not replace it. When the dashboard replaces the work, the work stops getting done.
Where the discipline actually shows up
The Copilot team did not invent some new optimization technique. They did the work of looking at full-task cost instead of per-call cost, then asking what they could change so the model finished the task with fewer redundant steps. Most of what they landed on was unglamorous. Each of those is a judgment call, not an algorithm. That is what makes them hard.
The reason this is worth stealing is that the discipline generalizes. Anywhere you have an LLM doing multi-step work, the discipline of asking what the model actually needs to finish the task applies. Most teams skip the question because the answer is rarely obvious. Sometimes the right answer is “more context, not less.” Sometimes the right answer is “let the agent take an extra step instead of precomputing a guess.” Sometimes the answer is “drop the optimization entirely because the optimization was not the bottleneck.” If you cannot defend a change in those terms, you do not understand the change well enough to ship it.
The smallest experiment worth running
You do not need GitHub’s benchmarking rig to test this on your own agent. Pick one thing your agent does every day that has a clear definition of done. Get a baseline measurement of how long it takes the agent to finish, end to end. Save that number. Then make one targeted change and run the same task ten times. If the new median is faster or cheaper, keep it. If the new median is worse, throw the change away.
The discipline part is the step most people skip. Most teams do not throw away the failed optimization. They leave it in a code branch and quietly merge it six months later when nobody is paying attention. That is how the dashboard slides. Reject the change. Write down why. Move on. Over a quarter, that habit gives you a quiet advantage that compounds.
Where this framing falls apart
I want to be honest about the limits. End-to-end task cost is a real metric, but it is harder to define than per-call tokens. What counts as a task. What counts as done. Who decides when the agent finished. If you do not have answers to those questions, the dashboard will quietly invent its own answers, and you will be back to optimizing a number nobody trusts.
A second caveat. Some per-call wins are still real. Trimming a 50,000-token prompt is not vanity if the model still finishes the job. The rule is “cut if it does not force the model to re-derive,” not “never cut.” Discipline, not dogma.
The third caveat is vendor risk. Most agent products still ship per-call dashboards because per-call is what their billing systems measure. Until the products ship end-to-end cost views, you have to validate the savings yourself. Treat the vendor’s numbers as a starting hypothesis, not a conclusion.
Trade-offs
- Task-level metrics are harder to define. Per-call tokens are easy. End-to-end cost requires you to define what counts as a task and what counts as done.
- Some per-call wins are still real. Trimming a giant prompt is not vanity if the model still finishes. The rule is “cut if it does not force re-derivation,” not “never cut.”
- The discipline is the moat. Anyone can copy a single trick. Few teams build the catalog of what works on their own workloads.
- Vendor dashboards will keep lying to you. Until products ship end-to-end cost views, treat per-call numbers as suspicious and validate them yourself.
What I would tell past me
Stop reading the dashboard, start reading the diff. If the diff is shorter and the task is faster, you are winning. If the diff is shorter and the task is slower, the dashboard is lying to you. The next time someone on your team proposes shaving tokens off a prompt, ask one question first. Will the model still finish the job without re-reading, re-running, or re-deriving anything it already saw. If yes, cut. If no, leave it alone. That single question has saved me from shipping at least three fake optimizations this year, and it will save you too.