I Built a Container From Scratch in Go, And It Actually Worked

Building a container runtime in Go sounds intimidating, but the core of it fits in a few hundred lines. I spent a weekend on it, and what I got was a working tool that runs an isolated process with its own filesystem, hostname, and process tree. Not Docker. Not Podman. A small, opinionated piece of code I can read top to bottom.

The point of this exercise is not to replace containerd. The point is to demystify what a container actually is. Once you have written the namespaces, cgroups, and rootfs setup yourself, the rest of the container landscape stops feeling like magic. You stop being afraid of the OCI spec, the runtime, and the layers in between.

Why bother writing one yourself

I had a specific reason. I was debugging a flaky CI job that ran inside a container, and the symptom was a process that refused to die when its parent was killed. Standard debugging tools were not telling me which namespace was holding the process alive. Reading kernel documentation about namespaces only got me so far. I needed to see the lifecycle in code.

Writing my own runtime solved two problems at once. It forced me to learn the syscall surface (the set of system calls a program can make) well enough to reason about it. It also gave me a stripped-down testbed where every line of behavior was mine, so I could bisect the issue by removing features one at a time.

The container landscape has a lot of moving parts: the daemon, the shim, the OCI runtime, the image format, the registry. If you only ever work at the top, you never see the seams. Building the bottom layer is the fastest way to learn where the seams actually are.

The architecture: namespaces, cgroups, rootfs

A Linux container is three things working together. First, namespaces (a kernel feature that gives a process its own view of system resources) make the process believe it owns the hostname, the process IDs, the network, and the filesystem. Second, cgroups (a kernel feature that limits and accounts for resource usage) constrain how much CPU and memory the process can use. Third, a rootfs (a directory that becomes the process’s “/” filesystem) provides a chrooted (a sandbox where the process sees a different root directory) environment to run in.

If you set up all three correctly and exec (replace the current process image with a new one) into a shell, that shell is what most people would call a container. There is no daemon. There is no image format. There is just a process tree rooted at your shell, isolated from the rest of the system.

In Go, the standard library does most of the heavy lifting once you know which syscalls to use. The syscall package exposes Unshare, Mount, and Chroot directly. The os/exec package gives you a clean way to spawn the child process. You wire them together with a config struct, a setup function, and a runner.

A minimal implementation

Here is the skeleton of what I ended up with. It is not production code, but it is the same shape you would build on top of.

package main

import (
    "os"
    "os/exec"
    "syscall"
)

func runContainer() {
    cmd := exec.Command("/bin/sh")
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWNS |
            syscall.CLONE_NEWPID |
            syscall.CLONE_NEWUTS |
            syscall.CLONE_NEWNET,
    }

    must(syscall.Sethostname([]byte("mycontainer")))
    must(syscall.Chroot("./rootfs"))
    must(os.Chdir("/"))
    must(syscall.Mount("proc", "proc", "proc", 0, ""))

    must(cmd.Run())
}

That is the core of it. The flags tell the kernel to give the new process a fresh mount namespace (so its mounts do not leak to the host), a fresh PID namespace (so it sees itself as PID 1), a fresh UTS namespace (so it can have its own hostname), and a fresh network namespace (so it does not see the host network).

A few details matter. The order of operations is important. Chroot before chdir. Mount /proc after the chroot so the new proc filesystem is visible inside. Use Unshare if you want to keep the parent process in the original namespaces and only isolate the child. Use Clone if you want the parent to share a thread of execution with the child.

Cgroups: where the real complexity lives

Namespaces are easy. Cgroups are where I lost an entire afternoon. The interface split between cgroup v1 and cgroup v2 is a source of pain. Most modern distros default to v2, but a lot of tutorials and Stack Overflow answers still target v1, and the file paths are different.

The basic idea is the same either way. You create a directory under the cgroup hierarchy, write the process ID into the cgroup.procs (or tasks) file, and write the limits you want into the resource-specific control files. For memory, that is memory.max. For CPU, that is cpu.max. For PIDs (limiting the number of processes), that is pids.max.

A cgroup v2 setup looks roughly like this:

mkdir -p /sys/fs/cgroup/mycontainer
echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs
echo 100M > /sys/fs/cgroup/mycontainer/memory.max
echo "50000 100000" > /sys/fs/cgroup/mycontainer/cpu.max

When the containerized process tries to use more memory than the limit, the kernel’s OOM killer (out-of-memory killer, which terminates processes when memory is exhausted) steps in. When it tries to use more CPU than the limit, the kernel throttles it. When it forks more processes than the limit, the fork is denied. None of this requires cooperation from the process. The kernel enforces the limits on its behalf.

The systemd-managed cgroup tree complicates this. On systems where systemd is the cgroup manager, the path is /sys/fs/cgroup/system.slice/mycontainer or similar. The systemd-run command is a friendlier wrapper, but if you want to set up cgroups from your own code, you have to know the layout. I ended up writing a small autodetect that figures out whether the system is using v1 or v2 and what the slice prefix should be.

The rootfs: pulling and unpacking

The rootfs is a directory that contains a minimal Linux filesystem. Most container images are layered filesystems, but a basic runtime only needs the final merged directory. For my purposes, I downloaded a Debian base image, untarred it, and pointed my runtime at the result.

A practical approach is to use debootstrap (a tool that builds a minimal Debian rootfs from scratch) to create a directory, then mount the layers as needed. For testing, I usually just use the host’s existing rootfs and skip the isolation that a full image provides. The point is to get a directory that has /bin/sh and a few libraries.

Pulling images in OCI format (the standard container image format) is doable in Go using libraries like go-containerregistry, but it is its own project. If you want a turnkey solution, link against runc (the reference OCI runtime) and let it handle the rest. If you want to learn, do the tarball and the layer application yourself.

What I learned, and what I would do differently

Three things stood out. First, the OCI runtime spec (the formal specification for what a container runtime must do) is shorter than I thought. It is around 50 pages and most of it is obvious once you have set up namespaces and cgroups manually. Reading it after writing the runtime filled in gaps I had not realized existed.

Second, the security model is more nuanced than “container equals isolation.” Namespaces give the process a private view of system resources, but if the process can escape to the host somehow, the host sees the same UID (user identifier) as the containerized process. The standard advice is to map the container’s root to a non-root UID on the host, and that advice matters.

Third, the seccomp (secure computing mode, a kernel feature that restricts which syscalls a process can make) layer is the one I would add if I were doing this for real. Without seccomp, a containerized process can call any syscall the kernel supports. A real production runtime applies a default seccomp profile that blocks syscalls known to be dangerous. It is not hard to add, but it is enough work that I left it for a follow-up.

Trade-offs

Building your own runtime is not a replacement for Docker or Podman. Here is what you give up:

No image registry integration. Pulling and pushing to Docker Hub is its own project.
No standard image format support. You handle a directory, not a tarball of layers.
No prebuilt seccomp or AppArmor (a Linux security module that restricts program capabilities) profiles. You write the rules yourself or skip them.
No cross-distribution testing. My code works on my Ubuntu 24.04 box. I have not run it on Fedora, Alpine, or NixOS.

The upside is that you understand the stack. The first time something at the container layer misbehaves in production, that understanding pays for the weekend you spent writing the runtime.

When to do this, and when to skip it

Build your own runtime if you are debugging a low-level container issue, writing a teaching tool, or just curious about how the stack fits together. Skip it if you need to ship a working product. The existing runtimes are battle-tested. Reinventing the wheel for production use is a recipe for surprise security bugs.

The code lives in a small repo. It is not a project I plan to maintain. It is a tool I use to teach myself how the system actually works, and that is a use case the existing ecosystem does not serve well.

Filed under: #development #docker #programming

I Built a Container From Scratch in Go—And It Actually Works