A green CI run tells you the tests passed. It does not tell you the agent understood the project. A recent New Stack write-up makes the gap sharper: it is now the rule, not the exception, when AI coding agents submit work to projects with real contribution policies. The mistake is not bad code. The mistake is a tidy diff that respects none of the boundaries the maintainers spent years drawing.
This kind of pattern reads like the obvious worst case: an agent handed a small bug fix produces a diff that compiles and tests cleanly, and quietly rewrites a file in a directory that should never be hand-edited. The mistake is easy to miss because the diff looks tidy. Below is what a maintainer who watches this happen consistently would write down, and what an editorial recommendation would look like.
A passing test suite answers a narrower question than most reviewers assume. It confirms the code behaved as expected under the tests that ran. It does not confirm whether the agent read the project policy, respected ownership boundaries, or avoided a directory the maintainers do not want touched. Those are different questions, and conflating them is the most expensive mistake a maintainer can make with agent-generated work.
What goes wrong on a quiet Tuesday
A human contributor who has not seen the contributing guide will usually pause and ask. An agent will not pause. It will produce a tidy diff that compiles, run the standard test command, and call it done. If the mistake repeats across files, the reviewer’s eyes glaze over and the diff lands.
Common things that get missed when an agent works fast:
- Files in a
generated/orvendor/folder that should not be hand-edited - Required changelog entries, migration notes, or release notes
- Public API contracts that need a maintainer’s sign-off
- Documentation files that travel with the code change
- Snapshot files, golden files, and fixtures that should be regenerated, not edited
The deeper problem is structural. A test pass and a policy check are different feedback loops. Tests fire in seconds. Policy fires in a human review that might happen hours later. The agent optimizes for the feedback loop that fires fastest.
Why agents skip policy
The failure mode is rarely refusal. It is silent assumption. The agent opens the repo, reads the obvious files, makes a sensible patch, and never surfaces the question of whether the patch belongs in this repo at all. The patch is locally correct and globally wrong.
Three things compound the problem.
First, the agent’s reward function is local. A good patch to the requested endpoint scores higher than a careful patch that respects a rule two folders away. The agent has no signal that a CONTRIBUTING.md rule applies unless something in the immediate task points to it.
Second, the rule files are easy to miss. A root CONTRIBUTING.md is one of many files. AGENTS.md, SECURITY.md, CODEOWNERS, .github/PULL_REQUEST_TEMPLATE.md, and the issue template each carry a piece of the picture. An agent looking only for the contribution guide will miss the rest.
Third, the feedback is asynchronous. A failed test stops the agent on the next attempt. A failed review stops the agent on the next pull request. Most agents will not backtrack to apply a process improvement they learned from a previous PR.
What the editorial recommendation points toward is a short, named, root-level file that names every other file the agent should read, in order. The agent does not have to discover the rule chain. The file tells it what the chain is.
Forcing the agent to ask before every edit would slow it to a crawl. The cleaner pattern is to ask only when the rule files contradict each other. For everything else, the agent moves on its own inside the scope the maintainer gave it.
What an AGENTS.md could look like
A short preflight checklist at the repo root, named in a way the agent can find, can replace the discover-the-rule-chain problem. A working draft for a small team would look like this:
Read these first: AGENTS.md, CONTRIBUTING.md, SECURITY.md.
Before any edit, list the affected files, APIs, tests, and docs in your response.
If a directory is auto-generated, do not edit it.
Dependencies, releases, and public API changes need my approval first.
Run the project's documented checks, not just npm test.
In your final reply, name every rule file you read and every check you ran.
If two rules disagree, stop and ask me.
The file should be short on purpose. A long policy becomes background noise. Place it at the repo root under AGENTS.md, and add a one-line link from CONTRIBUTING.md so a coding tool that only reads one of the two still finds the rules.
Two checks that earn their keep
A CI gate that fails the build when a file inside the generated directory is hand-edited. The cost is about 30 seconds per run. The benefit, in editorial terms, is that the agent learns on the first attempt instead of on the third review cycle.
A pull request template that asks for two specific things: the policy files the agent read, and the validation commands it actually ran. A free-text box would be weaker, but the template forces the answer into the diff where the reviewer can see it.
The other items the original checklist suggests (branch restrictions, code-owner rules for releases and security, dependency review requirements) are nice to have but rarely catch a mistake on their own. They cost maintenance time and have not yet paid for themselves in most small teams. Add them when the agent grows past the simple bug-fix work it is doing today.
A more honest metric than patch count
Teams often measure agent success by patch count: how many PRs landed this week, how fast did the agent produce them. That number misses the cleanup work the same PR created.
A better metric set: how many review cycles did the average agent PR take to land, how many times the agent stopped on its own to ask, and how many CI failures were caused by the agent versus caused by the human. The first number drops when the checklist is in place. The second rises, which is a good sign: the agent is asking instead of guessing. The third is roughly even in most teams, which tells you most CI failures are not agent-caused.
I would not measure success by how little I had to intervene. I would measure it by how often the agent produced a contribution that was easy to review, followed the rules, and did not create cleanup work for someone else.
What I would tell past me
Three things.
- A green CI run is the start of a review, not the end of one. The agent has proven the code compiled and the tests passed. It has not proven it read your project policy.
- Forcing the agent to plan first costs about ten seconds per task and saves the next three review cycles. The plan does not need to be detailed. It needs to name the affected files and the checks the agent will run.
- The most expensive mistakes are not the ones the agent makes. They are the ones a sleepy reviewer approves because the diff looks tidy. The checklist makes the policy work visible so the reviewer does not have to remember it.
Trade-offs
Adding process slows the agent down. That is real. A read-first, plan-first sequence adds a round trip and the agent cannot ship a diff while you are away from the keyboard.
Documentation has its own cost. A long AGENTS.md becomes noise. A short one gets read. The editorial consensus from the New Stack write-up suggests keeping the file to the mistakes the maintainers have actually seen, not every edge case.
The automation is not free either. CI gates add minutes to every run. Code-owner rules create more failure modes. Every gate has to be worth the maintenance.
Asking the agent to stop on every ambiguity would be the safest setting. It would also be the slowest, and a slow agent is one that maintainers stop using. The right balance, in editorial terms, is to reserve hard stops for changes that touch generated files, public APIs, releases, and security reports. The agent moves on its own for ordinary edits inside the agreed scope.
The migration from “agent ships the diff directly” to “agent ships a plan first” takes about a week to feel natural, based on the New Stack report’s account of teams that adopted it. A useful first step is to enable the policy for new contributions only and leave existing workflows alone, then expand the scope as the team learns what it actually catches.
If you maintain a project that receives outside pull requests, a short AGENTS.md is the cheapest guardrail you can add. If you maintain a private codebase with no outside contributors, the same file still helps, but you can lean harder on the plan-first workflow because the human reviewer is the same person who wrote the policy.
Bottom line
Before your next agent session, add one small guardrail. Write down the policy you wish the agent had read, make at least one dangerous change fail in CI, and require the agent to report which rules it followed. That will not remove every mistake, but it turns invisible assumptions into visible process.