A detached LUKS header bit me on encrypted ZFS, here is what I tried

I spent an entire Saturday trying to boot a fresh Arch install with an encrypted root, a ZFS pool on top of LVM (logical volume management, the Linux kernel’s framework for carving a disk into flexible partitions), a detached LUKS (Linux Unified Key Setup, the standard disk encryption layer in Linux) header, and the bootloader on a USB stick. The installer finished. Reboot did not. The kernel could not find /sysroot and there was no emergency shell because root was locked behind a password I had set but the initramfs (the initial RAM-based filesystem that loads before the real root, used to mount encrypted or remote filesystems) never asked me for. I had access to a live ISO and nothing else.

This is the kind of post you file under “I wrote this so I do not have to debug it again.” I am not the first person to hit it. I am not the last.

What the layout was supposed to do

The goal was a clean separation of concerns. The disk holding the encrypted root has no header at all. The header lives on a USB stick that stays in my desk drawer when the laptop is in my bag. Without the stick, the disk is just random bytes. With it, the disk unlocks, LVM opens the volume group, the ZFS pool imports, and the system boots.

Three layers, in order:

• The disk has a LUKS container. The LUKS header normally lives at the start of the same disk. In this layout it does not. The header is a separate file on a USB partition.
• Inside the LUKS container sits an LVM volume group. One logical volume is a swap partition. The other is a raw block device.
• On top of that raw block device lives a ZFS pool. The pool holds the root filesystem.

The boot partition and the EFI system partition (the small FAT32 partition the UEFI firmware reads to find bootloaders) are on the USB stick too. No part of the boot chain touches the encrypted disk until the kernel has the LUKS key material.

The plan was sane. Arch has supported this layout for years. The mkinitcpio hooks (mkinitcpio is Arch’s tool for assembling the initramfs; hooks are the scripts that decide which kernel modules and binaries get bundled in) for it exist. The instructions are written down. The problem is that none of the instructions assume you made the same small mistake I made.

The mistake, in one sentence

I told mkinitcpio to put the encrypt hook before lvm2, so the initramfs opened the LUKS container with the detached header from the USB stick, then tried to assemble LVM, then tried to import the ZFS pool. The order is correct. The hooks are correct. What I missed is that with a detached header, the encrypt hook needs an explicit --header argument pointing at the header file. Without it, the hook falls back to looking for the header at the start of the encrypted block device. There is no header there. The kernel waits forever for a passphrase prompt that the initramfs never renders.

That part I learned from a search result. The part that cost me another hour was diagnosing why the emergency shell never appeared.

Why the emergency shell never showed up

Arch’s initramfs is supposed to drop you into a root shell when mount fails. Two things kept that from happening.

• rd.break in the kernel command line forces a break. I added it. The initramfs broke into the shell, then the shell tried to chroot (change root, swap the current root filesystem to a different one mid-session) and the chroot failed because no real root existed yet. The shell I got was useless. No journal, no clear error, no way to remount because the encryption layer was still waiting on the header.
• ro on the kernel command line without a matching rw later in the boot chain means the real root mounts read-only and the system panics on first write. That is a separate issue. It just made the boot fail mode look scarier than it was.

Two mistakes stacked on top of each other. One was the missing --header argument. The other was that I had set a root password but the initramfs was never reaching the point where it would have asked for one.

How I eventually fixed it

The fix was two changes and a rebuild.

First, edit /etc/mkinitcpio.conf so the encrypt hook carries the header file path. The relevant line went from:

HOOKS=(base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 zfs filesystems fsck)

to:

HOOKS=(base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 zfs filesystems fsck)

That is a visual joke. The actual change was in the encrypt hook arguments below the HOOKS= line. The hook needs to know the LUKS header is not on the disk. The line that does the work in cmdline looked like:

cryptdevice=UUID=abcd-1234:systemVG:allow-discards header=/dev/disk/by-uuid/efgh-5678

The header= argument tells the initramfs where to look. Without it, the fallback is the start of the encrypted device. With it, the initramfs reads the header from the USB partition, prompts for the passphrase, and only then opens the container.

Second, regenerate the initramfs with mkinitcpio -P. The -P flag rebuilds every preset, which matters because Arch keeps a separate image for each kernel and each fallback. A partial rebuild leaves you with a half-fixed system.

Third, fix the bootloader. My systemd-boot entry was passing rd.break as a debug aid I had added the night before. Removing it cleared the read-only panic on the next boot.

Three changes. One file. One command. One bootloader entry. Total time to land them on a working system: about ten minutes, once I knew what I was looking for.

What I would tell past me

If I could send a message back to the version of me that started this install at 9 PM on a Friday, I would say four things:

• The detached header is a real layout. Read the Arch wiki page for it before you start. The wiki has the exact encrypt hook incantation. Mine did not match it.
• Set up the live USB with the same mkinitcpio.conf you plan to ship. Diagnosing this from a live ISO whose tooling does not match the installed system is a waste of time.
• Test the unlock path from the live USB before you reboot. Open the LUKS container with the header file by hand. Confirm the LVM volume group comes up. Confirm zpool import finds the pool. Each of those is one command. Skipping them is what cost me the Saturday.
• Keep the boot partition and the LUKS header on different partitions of the same USB stick. It is tempting to put them on different sticks. That looks safer. It is harder to keep straight, and it doubles the failure modes when one stick is missing.

Trade-offs

Operational cost is the first one to think about. The header lives on a USB stick. Lose the stick, lose the disk. There is no recovery path that does not involve a backup of the header file or a full re-install. For a laptop I carry every day, that tradeoff is acceptable. For a server in a rack, I would not consider it.

Install complexity is the second cost. mkinitcpio gets one more argument to get right. The systemd-boot entry gets one more variable. Each new layer is one more place to typo. A simpler layout, like putting the LUKS header back at the start of the encrypted disk, would have booted first try. The separation of concerns is the only thing I gained, and it is a real gain for a portable machine.

Debugging is the third cost, and the most painful one. When the layout does not boot, the failure mode is not a clean error. It is a kernel that sits forever on a prompt nobody can see, or an emergency shell that drops you into a chroot that does not work. Plan to debug from a live ISO whose tooling matches the install. That is the single most useful thing I learned this weekend.

Recovery time is the fourth cost. If the USB stick dies, I am not locked out of my data forever, but I am locked out for the length of time it takes to image a new stick and copy the header file onto it. A spare stick with the header pre-staged is worth the five minutes.

Bottom line

If you want a portable encrypted ZFS root with the LUKS header off the disk, this layout works. The Arch wiki page documents it. The mistake I made is the kind of mistake anyone could make: leaving the default encrypt hook arguments in place and assuming the detached header would be found by magic. It will not. Spell out the header file path. Rebuild the initramfs. Test the unlock from a live ISO before you reboot. Those four steps cover the entire failure mode I hit and the entire fix.

If you do not need the LUKS header off the disk, do not do this. The simpler layout will boot first try and you will not spend a Saturday on it.