Roughly one in seven CPU cores serving git.kernel.org exists for one purpose: rendering commit pages on behalf of software that never reads them. The Linux Foundation’s Konstantin Ryabitsev finally published the math behind the hunch every open source maintainer has carried for a couple of years. The bulk of inbound traffic at the kernel’s git host is not human, and most of the non-human share comes from AI vendors who reached for the most expensive possible way to gather material that was sitting in the open the whole time.
The headline number is about 258,160 CPU-hours per day, split across the 922 forks the kernel project publishes. That total is what pushed the maintainers to share the breakdown. It is not a rounding error, and the playbook that produced it is showing up at every public repository, public docs site, and open dataset on the internet. If your hosting bill has been creeping up while your user count stayed flat, this is the kind of traffic to check for first.
A polite download costs the host almost nothing
The whole story sits in one comparison. Walking the kernel repo’s full commit history through a normal git clone costs the server roughly 200 CPU-seconds. That is how git was designed to be used: one transfer, one small CPU burst on the host, and the client does the rest. The expensive alternative is to walk each commit’s HTML view one request at a time. The same 1.48 million commits that finish in three minutes via a clone take 280 CPU-hours when the server has to render every commit page on demand. The polite path comes out roughly 4.6 million times cheaper, and the bot operators know the math.
A few details worth pulling out of that gap:
- A clone is one round trip with a single short burst of work on the host
- An HTML walkthrough is millions of small calls, each one running git plumbing and a template render on the server
- The scraper saves a few engineer-days by skipping the work of writing a real git client
- The host pays the full price in CPU time, egress, and patience from the people who actually maintain the code
The scrapers are not picking the hard path by accident
This is the part that bothered me when I read the breakdown. The crawlers are not choosing the expensive route because they do not know git clone exists. They are choosing it because the per-request bill lands on the publisher, not on them. The crawler spends nothing on CPU for the render work, cycles through addresses from a residential proxy pool, and treats the origin server as a free resource to be drained. The same shape shows up everywhere open data lives: public APIs, docs portals, archive mirrors, dataset endpoints. The cost of being polite lands on whoever is fetching. The cost of being greedy lands on whoever is serving. Until publishers can flip that asymmetry, expect the drain to continue.
One signal that the cost asymmetry is intentional: the scrapers do not bother with caching, batching, or hitting public mirrors. They could cut their own bill by an order of magnitude with a few days of engineering. They do not, because the per-page revenue on the other side of the harvest is real and the per-page cost on your side is invisible to them. That gap is the whole opportunity.
Three generations of defenses, each one eventually broken
The kernel team has not been passive. They have cycled through three distinct waves of countermeasures, and each one bought real time before the bots adapted around it.
Fail2Ban plus IP-level blacklists worked briefly, until scrapers spread requests across whole subnets and forced the defenders up the stack. ASN-level blocks held up longer, until crawlers rotated through millions of residential and mobile addresses that no one can blacklist without also breaking legitimate visitors. Today the working defense is a proof-of-work challenge served by a tool called Anubis, which forces every visitor to solve a small CPU puzzle before any page renders. That slows bots down by making them spend CPU on their own side, but it also adds a click and a delay for first-time human visitors who have no idea why they are being asked to prove themselves.
Every defensive move raises the cost for everyone, including the audience you actually want to reach. Pretending otherwise is how you ship a website that nobody can read. The honest framing is that you are picking which visitors you want to lose, not whether you lose any.
What this means for anyone running a public service
If you host a repository, run a docs site, or expose any kind of API, you are next in line and the economics are not on your side. A scraper can spin up unlimited workers, rotate addresses, and absorb a proof-of-work challenge for pennies per request. You pay the full server cost on the other end. Open source works because the data is free to grab, and that openness is being turned against the projects that offer it. The kernel team can afford to keep the doors open and absorb the bill. Most smaller projects cannot, and they will quietly add logins, throttle endpoints, or shut down entirely. You will not see it happen, and you will lose projects you relied on.
A short list of signals worth grepping your logs for:
- The same handful of crawler identifiers dominating your access records at 3 a.m.
- Egress costs that grew this year even though your user count did not
- One or two endpoints that account for a wildly disproportionate share of CPU
- Crawlers that respect robots.txt only on the routes you actually check
If any of those match your setup, you are not imagining the slowdown. You are measuring it.
Trade-offs
Naming specific bots by user agent and ASN risks turning this into a company-versus-company story instead of a protocol-design problem. Pushing harder on proof-of-work adds friction to first-time visitors who cannot see why they are being asked to prove themselves. Charging AI vendors for API access sounds clean until you remember that academic researchers, indie developers, and small teams also depend on those crawlers and cannot pay enterprise rates. Every defense has a downstream cost, and the people paying it are rarely the people who caused the problem. The honest position is that open data should stay open, and the organizations consuming it at scale should pay in proportion to their use. Until the economics flip that way, expect smaller projects to quietly disappear behind logins that nobody asked for, and expect the whack-a-mole to keep going at the high end.
Coach’s note
If you run any public-facing service with crawlable content, the cheapest diagnostic you can run today is to sort your access logs by request volume per client. Look at the heaviest senders. If the well-known AI crawlers together account for more than a tenth of your traffic, you are already in this fight whether you noticed it or not. Decide now whether you want to filter them, slow them down, or shift the per-request cost back onto them with a challenge page. Waiting until your machine is pinned and your host emails you an overage notice is the worst possible time to make that call. Treat the kernel team’s data like the early warning it is, because the playbook that produced it is already heading for your server.
Source: https://feed.itsfoss.com/link/24361/17435468/linux-kernel-repo-ai-overrun
Verification token: EDLJX-20260902-125357-28446-1