>
Tech News

CERN trades Red Hat for Debian on 2,200 control machines

I have been a CentOS shop since before CentOS was a problem, and the kind of story CERN engineers told at MiniDebConf Winterthur in early 2026 hit close to home. They are not switching off Red Hat-derived distributions because Debian is cooler. They are switching because the math said one plan cost CHF 5.4 million with a one-in-five chance of working, and the other plan cost a smaller number with a much higher success rate. This is a story about what happens when a long-running industrial Linux environment has to keep running on a deadline, and the people running it actually did the homework.

The talk was given by Federico Vaga and Nikos Tsipinakis, who manage the control plane for CERN’s accelerator chain. The chain spans roughly 43 square kilometers and touches around 17,000 devices. Most of those devices sit in instrument racks next to magnets, detectors, and power supplies. None of them are general-purpose servers you can patch on a Tuesday afternoon and forget about. They are small industrial boards that run unattended for years and have to behave when somebody finally does look at them. When the team counted those machines, the answer was about 2,200 of them, and every single one needed a path off Red Hat before the end of 2026.

If you operate industrial hardware at any scale, here is what stood out from the talk.

  • The deadline was real hardware obsolescence, not a license change.
  • The decision took roughly two years of validation work, not a single executive order.
  • The deciding input was a risk-and-cost spreadsheet, not a feature matrix.
  • The hedge that made the switch possible was an abstraction layer above the operating system.

The Hardware Deadline Was the Real Driver

Most Linux migration stories start with a vendor pricing change or a support contract expiring. This one did not. The 2,200 control machines had a parts obsolescence clock. Boards CERN was using were going out of manufacture, and the replacements were not pin-compatible with the existing rack wiring. If the team stayed on the existing Red Hat derivative, they would have to redesign the boards they plugged into, redesign the rack interfaces those boards talked to, and then rewire and re-commission most of the 2,200 systems.

That is a lot of failure modes stacked on top of each other. Even in the best case, the engineers estimated they would succeed about 20 percent of the time. Not 20 percent of the way through the project, but 20 percent of the boards working after a full re-spin. With the accelerator’s Q4 2026 deadline, a one-in-five success rate was not survivable.

The team sized the cost of staying on the Red Hat-shaped path at around CHF 5.4 million. Eleven boards had to be redesigned. More electronics work, more software rewrites, and many more technician hours to rewire the racks. And every additional board you touched was another place the project could slip a quarter.

So they did the boring thing and treated the operating system as a software choice instead of a hardware one. Debian 13 could run on the existing hardware. The plan stopped being “redesign the boards” and started being “validate the OS upgrade and run the migration playbook.” Two very different budgets.

They Did Not Start From Scratch When CentOS Stream Broke

If you have been in this industry long enough, you remember the late 2020 announcement where Red Hat pushed CentOS from being a stable rebuild of RHEL to being a rolling-development branch upstream of RHEL. That move shortened the usable life of the entire Scientific Linux → RHEL → CentOS lineage by years. For most of us it was annoying. For a control plane like CERN’s it was a forcing function, because the cadence of upstream changes had a real chance of breaking an instrument run.

The team had already done one smart thing before that announcement. They had spent years making the integration layer distribution-agnostic. Their control software talks to the operating system through a thin middleware shim that does not assume a particular vendor’s tools. That shim was the only reason a Debian swap was even on the table. If the control code had been tightly coupled to RHEL conventions, switching distributions would have meant rewriting the whole stack, and they would have redesigned the boards instead.

A few things from that pre-work are worth copying if you are in a similar position.

  • Isolate your hardware abstraction in a layer that does not know which distro it runs on.
  • Run a second distro on a representative subset of your fleet for a year before you need to switch.
  • Track your board vendor’s parts-availability roadmaps on a calendar, not in a wiki page nobody reads.

The Migration Math, In Plain Numbers

The numbers from the risk analysis are the part of the talk I keep coming back to. They are not fancy. They are the kind of math any controls engineer can redo in a spreadsheet.

Staying on the Red Hat-shaped path required a CHF 5.4 million investment, eleven board redesigns, and an estimated 20 percent success rate against a fixed deadline. Switching to Debian required several years of validation work that had already happened, plus the cost of running a 2,200-system rollout with per-rack rewiring. The latter is not free. But every part of it has a defined engineering scope and a known success rate. The former has a defined budget and an unknown success rate, which is the worst possible combination.

The 2,200 machines are not all moving at once. CERN has staged the rollout so the most exposed racks migrate first and the well-understood ones go last. That is the only sensible way to do an industrial migration. Big-bang cutovers only work in software demos.

What You Would Copy From This

If you operate industrial hardware and you have not yet built a portable middleware layer above your operating system, do that before anything else. The CERN story is not about Debian versus Red Hat. It is about a team that made their control plane portable enough that they could pick a different OS on a Tuesday and still keep the beam on by Friday.

The second lesson is to do the cost math on the option you would rather ignore. The CHF 5.4 million figure is what changed the conversation at CERN. Without it, the team would have stayed on the Red Hat-shaped plan because that was the path of least resistance, and they would have discovered the 20 percent success rate at the worst possible moment.

The third lesson is that Plan Bs only work if you have actually tested them. CERN had Debian running on real systems while CentOS was still working. That is the only reason the Plan B was an option when the deadline arrived.

Trade-offs

Switching to Debian is not free. CERN spent years of engineering effort building and validating the portable middleware layer that made the swap possible. For a smaller shop without an accelerator to fund that work, the upfront cost can look worse than the alternative.

In our case, the trade was between paying CHF 5.4 million in board redesigns and technician time on a Red Hat-shaped path with a one-in-five success rate, versus paying for years of middleware engineering that turned the OS choice into a software decision. CERN picked the option that traded hardware redesign for software engineering hours. Your math will look different if you are a five-person team with no in-house distro expertise. Be honest about which line items you can actually cut.

The migration itself is staged, not a single weekend cutover. CERN sized it as a 2,200-system rollout with per-rack rewiring and re-commissioning. If you do not have the bench depth to validate each step, the same plan will not fit on your timeline. The shape works because CERN had the people. The shape does not work if you do not.

If your operating system is on a vendor-driven obsolescence path and you have a fixed deadline, building a portable middleware layer above the OS is a clear win. If your deadline is theoretical and your team is two people, the cheaper option is to ride out the existing platform, document the cutover path, and revisit when the deadline becomes real.

Bottom Line

If you are mid-migration off CentOS or any Red Hat-derived distribution, do not waste time on the RHEL-versus-Debian debate. Pick the operating system your team can actually operate, validate the move on a representative subset of hardware, and lock the timeline before the deadline starts moving. The CERN story is not about Debian being magical. It is about doing the boring work early so the interesting work still has room to happen.

Leave a comment