I lost a Saturday to a VM that was supposedly sized for the workload. Throughput had been flat for a week, latency climbed every afternoon, and the on-call dashboards told me nothing useful until I happened to run top on the guest and saw a %st column that was stuck around fifteen percent. The fix was not buying a bigger box. It was understanding that the host was oversubscribed and that the guest had no way to tell me.
A new Linux patch series aims to give guests that voice automatically, and it is the kind of change I want on my side of the fence even if I never run the hypervisor myself. Here is what is in the patch, what I would actually do today, and where I think it falls short.
What I would do right now
If you run any VMs and have not looked at %st recently, this is the only paragraph that matters today. SSH into a guest, run top once, leave it running, and walk away for a coffee. If %st is under two or three percent on a busy host, you are probably fine. If it is over five percent on a regular basis, you are paying for vCPUs you are not actually getting, and that is the number you want to track before you consider any of the rest of this article.
Track it the way you would track disk usage: write it down once, write it down again next week, and compare. The kernel does not give you a historical view by default. A simple cron job that runs vmstat 1 60 and writes the summary to a file is enough. That baseline is what you compare any future change against, including the patch described below.
A short checklist for anyone running a small fleet:
- Pick two representative VMs per host, not the busiest one
- Sample
%stonce an hour, for a full week - Compare guests that share a host, not guests across different hosts
- Note which workloads spike the number and which stay calm
That is the whole starting point. The patch is interesting, but it is a story for after you have a baseline.
What the patch is actually doing
Once you have a baseline, the patch story is easier to follow. A kernel developer has been posting revisions of a CPU-time-sharing adjustment targeted at guest kernels, and revision v11 is the one most people are talking about. The shape of the idea is small enough that I can describe it in a paragraph.
Inside the guest, the kernel polls its own CPU-deny count. When that number climbs past a threshold, the guest starts behaving as though the operator handed it fewer cores than it actually owns. The scheduler concentrates where it tries to place work, which cuts down on forced hand-offs and on time spent blocked behind synchronization primitives (lock variables that hold one worker while others wait to read or write shared state). When the deny count drops back to normal, the guest opens up again and uses the full vCPU allotment.
The whole thing is meant to be automatic. There is no resize script to run at 3 AM. The guest quietly contracts under pressure and re-expands when the pressure lifts. There is one big caveat, and I want to put it front and center. This is a patch series on a mailing list, not a finished feature in any released kernel. The proposal is targeted at the Linux 7.3 development cycle, with a real shot at landing in 7.4. If you want it in your data center today, you are building kernels and reading patch replies. If you want it on production guests, you are waiting several months.
Where this helps and where it does not
The governor is a guest-side change, and that matters more than it sounds. It reacts to the slice of CPU time the host decides to deny this VM. It cannot ask the host to be smarter about which other guests get priority, and it cannot detect contention on resources it does not measure. If your actual bottleneck is disk I/O on a shared LUN, or memory pressure from a sibling VM ballooning, no amount of CPU-steal awareness will fix it. Those are separate problems and deserve separate tools.
What the patch does well is the thing I have wanted for years: a guest that backs off gracefully instead of fighting the host for cycles it was never going to get. For mixed-tenancy clouds where the operator does not control every neighboring workload, that is a quiet win. For a single-VM home lab, the win is smaller but real.
For most teams running a mix of batch and interactive guests, this is a welcome addition once it lands. For teams running tightly-packed VMs that all need every cycle at the same time, the patch is closer to a wash. The interesting question is whether the maintainer adds an opt-out knob for the second group. I will be watching the next revisions.
Trade-offs
The steal governor is not free in effort. It shifts the cost of overcommitment from a noisy-neighbor blast radius to a quieter, per-guest CPU throttle. A VM that used to consume all 32 of its vCPUs at peak might intentionally leave the rest idle when steal time is high. For a workload that genuinely needs every cycle, that is a real loss. For a workload that was already suffering from contention, it is a clear win.
The behavior is also automatic, which is great until you want to override it. There is no knob in the patch series today for telling the governor “I know the host is busy, just give me everything anyway.” If your team has ever needed that escape hatch for a latency-sensitive job, you will want to read the follow-up revisions to see whether the author adds one. Keep an eye on the v12 thread once it lands.
If you only do one thing from this article, run top on a few VMs today and look at %st. The number tells you whether you have a real problem or just a paper one. The rest can wait until the patch lands.
If I could send a message back to the version of me that spent that Saturday chasing throughput on a supposedly well-sized VM, I would say three things.
- Trust the guest view, not the dashboard. Dashboards report what the operator thinks the guest has. The guest reports what it actually got. Those are different numbers when the host is oversubscribed.
- Measure before you resize. A week of
%stsamples is worth more than a guess about whether to add cores or move tenants. The number tells you whether you need more cores or fewer neighbors. - Patch your mental model of capacity. A 32-vCPU VM is not a 32-vCPU machine. It is a 32-vCPU promise, and the promise depends on what every other tenant on the host is doing. Treat vCPUs as a budget you spend against a shared pool, not as private cores you own. That shift in framing is what makes the rest of the diagnosis make sense.