>
Software

A recursive PHP script needs more than kill to clean up

What a runaway PHP loop actually leaves behind

A friend of mine once ran a small cron job that called itself through PHP’s exec() (a function that shells out to a subprocess and waits for it). The script was supposed to send a reminder email, fan out to a child process, and exit. It worked in staging. In production, it forked once, the child forked once, the grandchild forked once, and within fifteen seconds there were six thousand Apache httpd processes (the Apache HTTP Server worker processes that handle inbound requests) chewing up memory and sending the same email to a Gmail inbox roughly every two hundred milliseconds. The inbox went from empty to over limit in eleven minutes.

kill -9 did nothing. killall -9 httpd did nothing. The processes kept coming. The reason is that Apache mod_php spawns each request into its own worker, and a runaway script that calls exec() on itself is a new HTTP request from the local web server’s perspective. The web server is happy to spawn you a new one every time. Killing the visible processes does not stop the web server from accepting more requests that turn into more visible processes.

This is the part that catches new admins: kill -9 does not stop the cause. It stops the symptom that is currently visible. The cause is somewhere upstream, either the cron entry itself, the script being callable from the public web, or a worker that respawns after death.

What actually fixes a recursive PHP loop

There are four layers to this, and you usually need at least three of them. Picking only one wastes an afternoon.

  • Stop the producer. If the runaway came from a scheduled job, comment out the cron entry first. If the runaway is reachable over HTTP, block the URL path at the web server level before you do anything else. If the runaway is internal and only reachable from the script, find the script and remove or rename it. None of the fixes below matter if the producer is still alive.
  • Stop the web server’s worker queue. For Apache, that means apachectl stop or systemctl stop httpd. Killing the parent Apache process unblocks the accept queue (the queue of incoming HTTP requests waiting to be assigned to a worker). Workers die cleanly when they finish their current request, which for a php exec() infinite loop means they finish when they finish, not when you tell them.
  • Kill remaining workers. After the parent stops, list the still-running httpd processes with pgrep -a httpd. Most of them will exit on their own as their loop unwinds. The rest need a targeted kill -TERM (the polite “please finish and exit” signal), waiting thirty seconds, then a kill -9 for whatever is left. The TERM step is what most guides skip, and it is what causes the “I killed them but Apache still starts them up” confusion.
  • Clean up the side effects. Empty the Gmail inbox or set up a filter to drop the bounces. If the loop wrote anything to disk (database rows, log spam, file uploads), check the disk and the database for partial state. A run that has produced six thousand forked children has almost certainly produced half a million database rows too.

That is the sequence. Skipping the first two and going straight to killall -9 is the move that loses an afternoon. The producer is still running. The web server is still accepting requests. Killing the children does nothing to either of those.

Why pkill -f is the wrong reflex

The first instinct for many admins is pkill -f "php script.php" to match processes by their command line. It feels surgical. The problem is that on a recursive loop the script’s command line is exactly the same in the parent, the child, and the grandchild, and the respawned Apache worker holds the same command line for a few hundred milliseconds during startup. pkill -f kills the children but the next Apache worker respawn grabs the same script and keeps going.

The reliable signal for a runaway PHP loop is volume. A working PHP request lasts under a second. A recursive loop holds a worker for several minutes or until the script times out via max_execution_time (PHP’s per-script timeout, usually thirty seconds in production). After apachectl stop and the TERM window, anything still hanging around after sixty seconds is the genuinely stuck one. That is what gets kill -9.

What I do differently now

When I deploy any PHP script that calls exec() against itself, or against a URL on the same host, I now wrap the call in three guards:

  1. A file lock. Acquire an exclusive flock() on a sentinel file before launching the subprocess, and refuse to launch if the lock is held. This is one line in the parent and it kills the recursion at the very first fork.
  2. A process cap. Read /proc/<pid>/status and refuse to spawn a new child if the current PHP process count crosses a threshold. The threshold depends on the host, but on a 2-core VM with 4 GB of RAM, 30 is a reasonable cap.
  3. A mail throttle. Wrap the email send in a rate limiter. Even with the file lock and the process cap, a wrong setting on the cron can produce a runaway that lands twelve thousand emails before anyone notices. A simple per-minute counter inside the script keeps the blast radius under a hundred messages.

None of these guards solve the deeper operational question of “why is a script allowed to call itself recursively,” but they turn a five-minute problem into a fifteen-minute one, which is the difference between a recoverable incident and a coffee-spilling afternoon.

Trade-offs

There are real costs to the layered approach:

  • The file lock and process cap cost a few milliseconds per request. On a high-volume production system that runs millions of PHP requests per hour, that is measurable. The conventional answer is to put these guards only on the scripts that actually need them, identified either by code review or by the first incident.
  • Disabling exec() outright via php.ini‘s disable_functions is the cleanest defense, but it breaks everything from legitimate composer installs to WordPress update routines that shell out to git or wp-cli. The compatibility hit is large enough that most teams keep exec() enabled and put the guards inside the scripts.
  • apachectl stop (or systemctl stop httpd) during an incident takes the whole web server down. Other unrelated sites on the same host go dark too. For a single-tenant host, that is fine. For a shared host or a host with multiple Apache VirtualHosts (named-site blocks in the Apache config), this is a real cost. The alternative is to drop just the affected VirtualHost into a “503 Service Unavailable” via a .htaccess rewrite, which is faster but harder to revert cleanly.
  • The cap values (max_execution_time, the process cap of 30, the sixty-second TERM window) are all numbers I picked for one server. They will be wrong for yours. The trade-off is speed of recovery during an incident versus tuning time before the incident.

What I would tell past me

If you are reading this at the moment when your inbox is exploding and killall is not working:

  • Run apachectl stop first. Yes, the site goes down. That is the point. Half-measures leave the producer alive.
  • After thirty seconds, pgrep -a httpd | wc -l. The number that comes back is the genuinely stuck set. Compare it to your normal worker count. If it is ten times normal, the runaway is still spinning, and a second apachectl stop is not going to help; you need to actually find and kill the parent script.
  • Before you start any of this, check journalctl -u httpd --since "10 minutes ago". The access log will show the same URL hitting the server thousands of times per second. That URL is what you need to block at the firewall or the .htaccess level. The runaway will resume if you bring Apache back up without blocking the URL.
  • Do not be tempted to reboot the server. Reboot does not stop the cron entry that triggered the runaway. Within five minutes of the reboot, the same six thousand processes are back. Block the URL, fix the script, then reboot if you still want to.

PHP recursive loops look like an OS problem because the symptoms are at the OS level. They are a script problem that became an Apache problem because the script happened to be reachable from a public web path. Work the chain from the script back, not from the top down.

Filed under: #tools

Leave a comment