A migration that started with a toolchain decision
Engineers at CERN are pulling more than 2,200 industrial computers and embedded controllers off of Red Hat Enterprise Linux and onto Debian 13. The story broke at MiniDebConf Winterthur, in a talk by Federico Vaga and Nikos Tsipinakis, and the lessons travel well beyond particle physics. A lot of small teams run old hardware against a vendor’s latest compiler baseline and have not done the kind of risk analysis CERN did before the vendor’s roadmap forced the issue. This is a clean example of recognizing the trap early, putting real numbers on it, and choosing a target that fits the actual fleet.
CERN kept the scope tight on purpose. Only the accelerator control layer is changing, the front-end computers that talk directly to beam equipment, while the rest of the lab’s general-purpose IT stays on whatever it currently runs. That focused choice keeps the blast radius small if something breaks, and it lets the team validate the new stack on a contained fleet before touching anything mission critical.
The scale here matters. The 2,200 machines involved are spread across the accelerator complex, and the rollout order will track the maintenance windows already on the accelerator calendar. That last detail is the kind of thing you only see when an engineering team writes a real plan instead of just deciding on a direction.
The compiler flag that started this
What pushed CERN off Red Hat was not a missing feature or a security scare. It was a single decision in Red Hat’s build process about how old a CPU the binaries will still support. Each major RHEL release draws a line somewhere on the silicon timeline, and that line ends up baked into the compiler invocation. RHEL 9 already drew the line above CERN’s oldest boards, including the machines running chips from the Core 2 Duo generation, which together are nearly half of the accelerator fleet. RHEL 10 moved the line again, this time pushing even Ivy Bridge era boards out of the supported window. That second cut is what made staying on Red Hat untenable, because it was no longer a question of supporting one generation of old chips but of rewriting for two generations at once.
The cost of trying to stay was not small. CERN’s own estimate put the rewrite bill at roughly CHF 5.4 million. The hardware side alone meant eleven boards needing a full redesign, with racks needing to be rewired and most machines impacted during the commissioning phase. The team estimate for the rewrite was six new hires: a pair of electronic engineers, a pair of software engineers, and a pair of technicians dedicated to the migration. Even assuming all of those replacements went in bug-free, the lab’s own probability estimate for the whole effort was twenty percent.
Why Debian fit the brief
Once CERN decided to leave, the question was where to land. Debian won because it matches the support window the hardware demands. The official Debian LTS track gives five years of security coverage for a release, and Freexian, a small Debian-focused services company, sells an extended program on top of that for older releases the foundation itself has retired. Together the two cover more than the decade the accelerator fleet needs. CERN has gone further and is now one of Freexian’s paying sponsors. That is both an endorsement and a hedge: if Freexian runs out of money tomorrow, the runway gets thinner fast and the plan has to be redrawn.
The rest of CERN’s IT infrastructure is staying on whatever it currently runs. This is not a wholesale desktop or server swap. It is a focused, surgical migration that targets the front-end layer where the hardware age problem actually hurts, with the rest of the lab’s general-purpose systems unaffected.
The two release paths CERN weighed
For keeping Debian in step with the maintenance schedule the accelerator already runs on, CERN weighed two paths. The first path develops on Bookworm (Debian 12) through 2026, deploys Trixie (Debian 13) as the long-term support release from 2026 to 2030, then jumps to Debian 15 once it ships. The second path skips the last jump and stays on Trixie under extended support out to 2033 instead. A third option, keeping Bookworm under extended support for the whole window, got ruled out during planning.
Both viable paths assume Debian’s LTS and ELTS programs stay healthy for the next eight years, which is part of why CERN is sponsoring Freexian in the first place. If Freexian went away, the runway would shrink and the choice would have to be revisited.
The boot stack that replaced tftp
What used to be NFS plus tftpd at boot is now a different shape. The old design, dating back to roughly 2005, handed a single image to every box and trusted the network. The new design pulls the bootloader, the kernel, the init RAM disk, and the userspace apart into independent layers, then orchestrates their delivery through Kubernetes. A reconciliation controller watches for drift between the intended configuration and the actual state on each machine, and triggers a redeploy when the two fall out of sync. That is the same pattern you would build for any modern fleet: declarative spec, controller loop, automatic convergence. The interesting part is not the new tech, it is that the new stack makes drift visible instead of waiting for a hardware tech to notice a stuck box at 2am.
Trade-offs worth naming
Even with the upside, the CERN team is taking on real new risk. A decade of accelerator uptime now depends on whether Debian LTS keeps its five-year promise, whether Freexian stays solvent enough to extend that another five, and whether the company’s ELTS arm keeps paying its maintainers. The migration also asks the team to operate Kubernetes at a scale they did not run before, with a reconciliation loop whose blast radius, when it fails, is the whole fleet at once instead of one machine at a time. The bill for staying on Red Hat was five and a half million Swiss francs and a one-in-five chance of getting all the rewrites right. The bill for leaving is operational complexity and a long-term bet on a smaller ecosystem. Both are real costs. The team picked the second one, and the planning numbers they published are what made that defensible.
What this teaches the rest of us
Most teams who find themselves in a position like CERN’s did not start there. They started with one good reason to standardize on a vendor’s platform, and the platform got more expensive on a schedule they did not pick. The expensive part was never the next subscription bill. It was discovering, two release cycles in, that you have to hire people and rewrite hardware just to stay where you are.
A short checklist for anyone in a similar spot:
- Write down what your oldest supported CPU generation is, and what the next vendor release will cut off.
- Estimate the cost of staying, including the people you would have to hire.
- Estimate the cost of leaving, including the new operational habits you would have to build.
- Pick the destination that fits your support window, not the one with the best marketing.
What CERN bought itself by switching is not really “Debian.” It is the right to make its own decisions about hardware, hiring, and the upgrade clock. That is a much more valuable thing than it sounds. Watch the MiniDebConf Winterthur talk recording when it shows up online, and if you are sizing up a similar move for your own fleet, get pricing from Freexian before you commit. LTS alone is rarely enough when the hardware is older than the support window.