Bootc and OSTree: Modernizing Linux System Deployment

Bootc and OSTree: how Linux system deployment is finally getting modernized

I have been running Fedora Silverblue (a variant of Fedora that uses an immutable (read-only) operating system image, updated as a whole rather than package-by-package) on my main laptop for about a year. Before that, I ran regular Fedora for about six years. The transition was not painless. The immutable-base model, where the operating system files are read-only and you layer your changes on top, was unfamiliar. The tool I underestimated was the one underneath both of them: bootc (a tool that builds and deploys container images as your operating system, the same way you deploy a container to a server).

bootc is the deployment story for the next generation of immutable Linux distributions. It is built on top of OSTree (a tool that versions the entire filesystem as a Git-like tree, letting you roll back the whole OS to a previous state) and on top of the same OCI (Open Container Initiative, the standard format used by Docker and Kubernetes) container images that the rest of the cloud is built on. If you have ever built a container image, you already know most of the deployment model. If you have not, this is a good time to learn, because the model is taking over.

The model, in one paragraph

Here is the mental model. Your operating system is a container image. The image is built the same way you build any other container image: a Containerfile (the container equivalent of a Dockerfile, listing the base image and the commands that customize it) that starts from a base image and adds your customizations. The image is pushed to a registry (a server that stores and distributes container images, the same kind Docker Hub and GitHub Container Registry are). The image is pulled onto your machine and “booted” as the running operating system. Updates are a pull of a new image. Rollbacks are a re-boot of the old image. The whole filesystem is versioned through OSTree underneath, so the disk state matches the image you booted.

The reason this is a big deal is that it solves three problems that have been open in the Linux desktop for a decade.

• Reproducibility. The image you build is the image that runs. There is no “works on my machine” problem, because the image is the machine.
• Atomic updates. Updates are applied as a whole. If the update fails to boot, the system rolls back to the previous image on the next boot. There is no “I updated my packages and now X is broken” failure mode, because the broken state was never the running state.
• Unified workflow. The same Containerfile syntax that builds a server container builds your laptop. The same registry that hosts your server images hosts your laptop image. The same CI (continuous integration, automated build and test pipelines) pipeline that builds your application can build your OS.
• Better hardware support at boot. Because the image includes the kernel and the firmware, the boot process is more reliable across hardware. I have noticed this on a few laptops where the rolling kernel of my previous install occasionally failed to load WiFi firmware. The image-based install does not have the same problem, because the firmware is in the image.

None of these are new ideas. ChromeOS has had all three for a decade. macOS has had atomic updates for almost as long. Linux is finally catching up, and the reason is bootc.

What the workflow actually looks like

I am going to walk through the workflow I have been using, because reading about it is not the same as seeing it.

The first step is to write a Containerfile. Mine is short.

FROM quay.io/fedora/fedora-bootc:41

RUN dnf install -y zsh neovim kitty syncthing && dnf clean all

COPY etc/ /etc/

RUN systemctl enable syncthing

The FROM line is a bootc-aware base image. The RUN line installs my custom packages. The COPY line copies my configuration files into the image. The systemctl enable line enables a service. That is the whole customization. The image builds with podman build (or docker build), the same way any other container image builds.

Next, push the image to a registry. I use quay.io (Red Hat’s container registry, similar to Docker Hub). The push is a single podman push command.

Finally, install the image on a machine. The bootc command on the machine is bootc switch quay.io/myuser/myimage:tag. The command downloads the image, lays it down on disk using OSTree, and reboots into it. The whole process takes about 5 minutes on a fast network.

That is the workflow. Build, push, switch. The same workflow I use for a web service is the workflow I use for my laptop.

What I had to relearn

I am not going to pretend the transition was smooth. There were three things I had to relearn.

• The package manager is still there, but it is for the image, not the running system. When I dnf install something in a running bootc system, the change does not survive a reboot. The change goes into a transient overlay that is discarded. To make a change permanent, I have to add the package to the Containerfile and rebuild the image. This is the right model. It took me about three weeks to stop typing sudo dnf install reflexively.
• The home directory is still mutable. The model is “operating system is immutable, user data is mutable.” My home directory is on a separate partition that is preserved across image switches. This is the right design. It means I can roll back the OS without losing my files.
• System configuration is now declarative. I used to edit /etc/ssh/sshd_config by hand. Now I edit the file in my Containerfile build context and rebuild the image. The benefit is that my configuration is in version control, in a Git repo, with a history. The cost is that I have to rebuild and switch images to change a single line of config.

The third one is the cost I underestimated. The first two are pure wins. The third one is a trade I have decided to make because the version control benefit is real.

What I would tell past me

If I could send a message back to the version of me that was running a traditional mutable Linux distribution, I would say three things.

• Start with a fresh install, not an in-place upgrade. The bootc model assumes a clean filesystem layout. Trying to convert an existing install is a weekend I would not get back. A fresh install is faster and ends in a system that actually works.
• Put your customizations in a Git repo from day one. The Containerfile, the etc/ overlay, the list of packages to install. All of it should be in version control. The first time you roll back a change, you will be glad you did this.
• Do not try to make the running system mutable. The temptation to sudo dnf install something and have it stick is real. Resist it. The whole point of the model is that the running system is not the source of truth. The image is.

Trade-offs

The rebuild-and-switch cycle is slower than dnf install for a single package. I have measured it. A single package change takes about 4 minutes to rebuild the image and 5 minutes to switch to it, on my hardware. The 9 minutes is the cost of atomic updates and version control. I think the 9 minutes is worth it for the kinds of changes I make once a month. I do not think it is worth it for “I need to test this one thing right now.”

Container registry dependency is a real one. I use quay.io. The free tier is enough for personal use. If quay.io goes down, I cannot rebuild or switch images. The mitigation is that the image on my local machine still works. I just cannot push a new one. The mitigation is not a perfect one. The dependency is real.

Community around bootc is small. The documentation is good. The number of people who have answered a question about bootc in a forum is small. If you are the kind of person who learns by reading other people’s solutions, you are going to do more reading and less skimming than you are used to.

Bottom line

bootc is the most important change in Linux desktop deployment in a decade. The model is not new. The implementation is finally good enough to recommend to someone who is not already a container expert. If you are running Fedora, Nobara, or one of the other bootc-based distros, the model is already underneath you. If you are not, this is a good year to try.