I started with a single Raspberry Pi running Docker Compose. One YAML file, a few containers, everything worked. Then I added a second Pi for redundancy. Then a third for more services. Suddenly I was SSHing into three different machines, manually copying YAML files, and praying that my “backup” rsync script (a tool that synchronizes files between machines) had actually run last Tuesday. I was the on-call sysadmin for a small business I had accidentally created, and I was bad at it.
Docker Compose (a tool that lets you describe a multi-container application in one YAML file and start it with one command) is wonderful for one box. It is less wonderful when you have three boxes, six services that need to talk to each other, and a desire to sleep through the night. This is the story of what I moved to, what I tried that did not work, and what I wish I had known at the start.
The first thing I tried: keeping Compose, just on three boxes
The low-effort path was to keep using Compose and just sync the YAML files between machines. I wrote a small make deploy target that ran rsync over SSH and did docker compose up -d on the remote. It worked. It also broke constantly. The state drifted because the three machines were not identical (one was a Pi 4, one was a Pi 5, one was an old Intel NUC). The YAML files grew small differences per host. The “deploy” step took longer than the development step. I gave up on it after about three months.
The deeper problem was that I was using files to model what was really runtime state. I had a Postgres container on the Pi 4, and I wanted it to fail over to the Pi 5 if the Pi 4 went down. Compose has no concept of “this container should be on exactly one of these three hosts.” It just runs whatever is in the local YAML. I could have written a custom failover script, but I knew that path ended in me debugging cron at 2 AM. I started looking at orchestration tools.
Kubernetes was the obvious answer. It was the wrong answer.
I had used Kubernetes (a container orchestrator that automates deployment, scaling, and failover of containerized applications) at work. I knew it well. I also knew it was overkill for a homelab with three machines. The minimum viable K8s cluster takes about a day to set up correctly. The maintenance burden is a part-time job. The resource overhead is non-trivial. I tried anyway, for about a month.
The cluster came up cleanly. The services deployed cleanly. The DNS (Domain Name System, the system that translates hostnames like home.lab into IP addresses) was a little fiddly. The storage story (how to keep data alive across container restarts and host failures) took me two weekends of fiddling with local-path-provisioner and rook-ceph. The networking took another weekend. By the time I had a working cluster, I had spent about 30 hours and I was not enjoying it.
The breaking point was a routine upgrade. A new version of one of my services required a newer version of the container runtime. The runtime upgrade required a node reboot. The reboot triggered a pod reschedule. The reschedule hit a bug in the storage class. The bug ate two days of Postgres data. I restored from backup, swore quietly, and tore the cluster down. Kubernetes is a real answer for real production workloads. It is not the sensible default for a homelab where I want to tinker in the evenings and not be on call.
What actually worked: Nomad with a small twist
HashiCorp Nomad (a scheduler that runs containers, virtual machines, and standalone applications across a cluster of machines, simpler than Kubernetes) was the tool I had been looking for. The mental model is closer to “systemd for a cluster” than “Kubernetes for nerds.” A single binary runs on each host. One of them is the server, the others are clients. You submit a job spec (which is just a HCL file, HashiCorp Configuration Language, similar to JSON but more readable) and Nomad schedules it.
The thing I appreciate most is how little ceremony there is. The first cluster came up in about an hour. The first service (a small static site) was running in 10 minutes. The second service (a Postgres instance) took an evening, mostly because I was being careful about backups. By the end of the first weekend I had three hosts running, services distributed across them, and a single nomad status command that showed me the state of everything.
The failover worked the way I wanted it to. When I pulled the power on the Pi 4, the services that were pinned to it were rescheduled to the Pi 5 within about 30 seconds. The Postgres container came up from its existing data volume. The static site (no state) was a non-event. The only thing I had to do manually was update a DNS record, and that took me about 10 minutes of looking at the docs.
The thing I added: Consul for service discovery
Nomad handles scheduling. It does not, by itself, know that service-a should talk to postgres.example.internal instead of a hardcoded IP. For that I added HashiCorp Consul (a service mesh, a network layer that handles service-to-service communication, including discovery, health checking, and encryption). It was the same install pattern: a single binary on each host, one is the server, the others are clients.
Consul gives you a DNS interface. When one of my services wants to talk to Postgres, it looks up postgres.service.consul and Consul returns the IP of whichever host is currently running the Postgres container. If the container moves, the DNS answer updates automatically. The TTL (Time To Live, how long a DNS answer is cached) is short enough that the service does not see stale IPs in practice.
I was worried about the operational complexity. In retrospect, it was about 20% more complex than running Nomad alone. The extra complexity is worth it because the alternative is a brittle make deploy script that has to know about every host.
What I would tell past me
If I could send a message back to the version of me that wrote the first Compose file on the first Pi, I would say three things.
- Do not write a custom failover script. I did. It was the worst code I have ever shipped. Use a real scheduler. Nomad is the one that worked for me, but K3s (a lightweight Kubernetes distribution designed for small clusters) would also have been a fine choice if I had been willing to accept the K8s overhead.
- Backups are not optional. I lost two days of Postgres data because my backup was running on the same machine as the database. Move backups off-host. Verify them. Test the restore. A backup you have not restored is a hope, not a backup.
- Service discovery is a real problem at three hosts, not one. A single host does not need Consul. Two hosts barely need it. Three hosts need it because the third host is the one that will be down when you are not home.
- Time spent on a real scheduler pays for itself the first time a host dies. The hours I would have spent writing failover scripts, backup verification, and ad-hoc service discovery are a small fraction of what I lost to the K8s detour. Choose the boring tool. Use it for a year. Re-evaluate then.
The order of operations that has worked in my testing is: get Nomad running with one job first, get a second host added second, add Consul third when you have a service that needs to find another service by name. Skipping straight to K8s because you read about it on Hacker News is a mistake I made. You do not have to make it too.
Trade-offs
Nomad is not free in time. The initial cluster setup took me about 8 hours across two evenings (install Nomad on three hosts, configure ACLs (Access Control Lists, a list of rules that say who can do what), set up a shared filesystem for stateful workloads, write a small script to back up the Nomad server state). The Consul install added another 3 hours. The first service I migrated from Compose to Nomad took about 2 hours, mostly because I was being careful.
Maintenance is also not free. Every host that joins the cluster needs a small amount of configuration. I have a bootstrap.sh script that I run on a fresh machine, and it takes about 10 minutes. I run it about once a quarter, when I replace a dead Pi.
The big upside is that I sleep through the night now. I used to have a 2 AM page in my head that said “is the homelab on fire?” I do not have that page anymore. I run nomad status in the morning, and if something is wrong, I see it. The other upside is that I can add a fourth host in about 15 minutes, and the services rebalance themselves.
If you are running one machine and one service, stay on Compose. If you are running two machines and three services, you are probably fine with rsync. If you are running three or more machines and you find yourself writing if statements in shell scripts to handle host failures, you are ready for Nomad. If you are running ten or more machines and a multi-tenant workload, you are ready for K8s, and you should not be reading a homelab blog for advice.
Bottom line
Three machines, six services, one person. That is a homelab that has outgrown Compose. Nomad and Consul are what I would call the sensible default for me. K3s would also be a fine choice for someone who knows K8s. A custom shell script is not the answer for anyone. I learned that the hard way, and the 30 hours of K8s I burned through were the tuition.