The 5-Minute Kubernetes Cluster Health Check

Kubernetes (the open-source container orchestrator that runs and schedules your applications across a fleet of machines) is great until it is not. One bad node, a single OOMKilled pod, and the rest of the cluster can spend the next forty minutes pretending everything is fine while your pager lights up. The reason a five-minute check matters is that the dashboard tells you the cluster is “Ready” right up until the moment it tells you it is “NotReady,” and the gap between those two states is where the outages live.

I run three small clusters for side projects and one larger one at work, and the routine below is what I run on a Monday morning, before any deployment, and when a colleague says “the staging cluster feels slow today.” It is a triage pass that catches about eighty percent of the problems I have actually hit. The other twenty percent need kubectl describe, log spelunking, and occasionally a phone call to the platform team. Those cases start with the same five commands, then go deeper.

Why a 5-minute check beats a 4-hour outage

The cluster health page in most dashboards reports a green status that is technically accurate and practically useless. Eight pods scheduled, eight pods running, zero restarts in the last hour. Green. The actual cluster could be running on a single node with the others in a flapping state, and the dashboard would still say green, because the running pods are running. What the dashboard does not surface is the second-order signal: a node that joined the cluster but cannot run new workloads, a pod that is CrashLoopBackOff but is being restarted quickly enough to count as “running,” a control plane component that is using 90 percent of one CPU.

The five-command routine I run is the cheapest way to surface those signals before they turn into a 3 AM page. None of the commands is novel. Anyone who has been on call for a Kubernetes cluster has run some version of all of them. The point of doing them in the same five-minute window, every Monday, is that the trends become visible. A spike in node memory pressure is a hint. A node that has been NotReady for ten minutes and then rejoined is a louder hint. None of these are visible if you only look when something is on fire.

The five commands, in order

These are the commands I actually type, in the order I actually type them, with the one-line interpretation that turns each command’s output into a decision.

• kubectl get nodes -o wide (lists every node in the cluster with its status, age, and resource version). The status column should be Ready for every row. Anything else is the start of the investigation. Pay attention to the AGE column. A node that joined two minutes ago after a reboot is not the same as a node that has been up for sixty days. Both can be Ready, but only one is trustworthy.
• kubectl top nodes (shows current CPU and memory usage on each node, requires the metrics-server add-on to be installed). Anything above 80 percent sustained is a problem. Anything above 90 percent is a problem that is about to become a worse problem. The numbers are not exact, but they are accurate enough to catch the difference between “fine” and “about to fall over.”
• kubectl get pods -A --field-selector=status.phase!=Running (lists every pod across all namespaces that is not in the Running phase). Empty output is the goal. Anything in Pending for more than a minute is a scheduling problem. Anything in CrashLoopBackOff is a code or config problem. Anything in ImagePullBackOff is a registry or credential problem. The phase tells you which category to investigate, and the namespace tells you which team to call.
• kubectl get events -A --sort-by=.lastTimestamp | tail -30 (shows the thirty most recent cluster events across all namespaces, sorted by when they happened). You are looking for repeated warnings about the same node, pod, or volume. A single warning is noise. The same warning three times in five minutes is a signal. The events view is the only place that signal lives, because the pods involved may have been recreated between the warning and your look.
• kubectl get pods -A -o json | jq '.items[] | select(.status.containerStatuses[]?.restartCount > 5) | {ns:.metadata.namespace, name:.metadata.name, restarts:.status.containerStatuses[].restartCount}' (lists every pod whose containers have restarted more than five times, with namespace, name, and restart count). Anything here is a code-level problem, not a cluster-level problem. The pod is alive enough to keep restarting. The cluster is doing its job. The container is not.

What each command catches that the others miss

The reason for five commands instead of one is that no single command surfaces every failure mode. A node can be Ready while a pod is unschedulable because of a taint (a marker on a node that tells the scheduler to keep certain workloads off it, which can be removed with a toleration) the cluster applied ten minutes ago. A pod can be Running while its container has restarted forty times. The events view can be empty while a HorizontalPodAutoscaler is thrashing. Each command catches a different class of problem, and the union is wider than any individual check.

• Node status (get nodes) catches the cases where a node has been removed from the load balancer pool (a configuration that tells your ingress, the component that routes external traffic into the cluster, which backend nodes are eligible to serve requests) but is still listed in the cluster. The dashboard does not surface this. The pod list does not surface this. Only get nodes does.
• Node resource usage (top nodes) catches the case where a node is technically ready but cannot take new workloads because it is already at 95 percent memory. The cluster reports green because the nodes are ready and the pods are running. The next deploy will fail to schedule, and you will not know why unless you checked the numbers.
• Non-running pods (get pods -A --field-selector=status.phase!=Running) catches the pods that did not start, the pods that crashed before they finished initializing, and the pods that are stuck waiting on a dependency. The running-pod view hides all of these by definition.
• Events (get events) catches the transient warnings that did not leave a pod behind. A failed liveness probe (a periodic check that asks “is this container still healthy?” and restarts the container if the answer is no) can succeed on the second try and leave no pod-level signal. The events view keeps the history.
• Restart counts (the JSON query above) catches the pods that are running but should not be. A pod with fifty restarts is not healthy. The phase view calls it Running. The dashboard calls it green. The restart count is the only place the truth lives.

What I do when something looks off

The five commands are triage. When one of them flags a problem, the next step depends on which one flagged. The patterns I have learned to recognize:

• A node in NotReady. Run kubectl describe node <name> and look at the conditions. The most common cause on my clusters is a kubelet (the per-node agent that talks to the control plane and runs the pods) that lost its connection to the API server. The fix is usually a systemctl restart on the kubelet, then a five-minute wait to see if the node rejoins.
• A pod in Pending. Run kubectl describe pod <name> -n <namespace> and look at the events at the bottom. The most common cause on my clusters is a missing PersistentVolume (a piece of persistent storage that survives pod restarts, requested by the workload via a PersistentVolumeClaim) or a node selector that no node matches. The fix is either to provision the volume or to relax the selector.
• A pod in CrashLoopBackOff. Run kubectl logs <pod> --previous (the --previous flag is the key, it shows the logs from the container’s last failed attempt rather than the current one). The cause is almost always a missing environment variable, a bad config file, or a database that the pod cannot reach.
• Repeated warnings in events. Look at the source component and the involved object. If it is the kubelet on the same node three times in five minutes, the node is the problem. If it is the scheduler for the same pod three times in five minutes, the pod spec is the problem.

Trade-offs

A five-minute routine is not free. The cost is the time itself, plus the habit of actually doing it. The first time I tried to make this a Monday-morning ritual, I missed three weeks in a row. The cost of not doing it is also not free. The first time I caught a flapping node at 9 AM on a Monday instead of at 2 AM on a Friday was worth about three hours of sleep.

Three honest costs to weigh:

• Tooling overhead. kubectl top nodes requires the metrics-server add-on. If your cluster does not have it, install it first. Without it, you are running four commands instead of five, and you are guessing about resource pressure. The install is one line of kubectl apply with a manifest from the Kubernetes project, and it takes about two minutes.
• Context switching. Five commands across four views is enough mental load that you will not retain the pattern unless you make it a habit. The first month is the worst. By month three, the sequence is muscle memory. By month six, skipping it feels wrong.
• False positives. The events view is noisy. A single warning about a failed liveness probe is usually a transient blip. You will spend your first few weeks chasing ghosts. The fix is to look for repetition. The same warning once is noise. The same warning three times in five minutes is the signal you came for.

For a single-cluster team, this routine takes about five minutes once you have the commands memorized. For a multi-cluster team, scale it to about two minutes per cluster if you have a script that loops through the contexts. The migration cost is one afternoon of writing the script and one week of building the habit. The hard part is the habit, not the script.

If you run Kubernetes in production, this is the routine I would build first. If you only run it for side projects and do not have a pager rotation, the value is lower but the routine still catches the problems that would otherwise eat your weekend.

Bottom line

If you only do one thing from this article, set up a Monday-morning five-minute check on your cluster. The other four days of the week can wait. The reason Monday matters is that Monday is the first day most people are paying attention, and the rest of the week inherits whatever state the cluster is in at 9 AM on Monday. A five-minute check on Monday is the cheapest insurance you can buy against a Friday outage.

Three things to keep in mind:

• The order matters. Nodes first, pods second, events third. Skipping ahead to events is tempting because events are more interesting, but you cannot interpret an event without knowing the node and pod state. Run them in order.
• The five-command set is a triage, not a fix. When something is wrong, you will reach for kubectl describe and kubectl logs --previous. Those are the next two commands in the runbook, not in the routine.
• Run it on a schedule, not on demand. The habit is the value. On-demand checks only catch the things you already suspect. Scheduled checks catch the things you did not.