Managing Services in Linux: The Practical Guide

Linux service management has two parallel systems, and the fact that both exist on most modern distros is a source of constant confusion. Systemd and SysV init do the same job in different ways, and most distributions ship with one of them plus a compatibility layer for the other. If you have ever wondered why your script works in a Docker container but not on a server, or why a service starts on one machine and not on another, the answer usually comes down to which init system is in play.

This guide is the one I wish I had when I started administering Linux boxes. It covers how services work, how to manage them, and how to debug the common failure modes. It is not a comprehensive reference. It is the practical subset you actually need.

The two systems, briefly

SysV init is the older approach. Services are shell scripts in /etc/init.d/. They get started at boot by symlinks (symbolic links, basically shortcuts) in /etc/rc?.d/ that point back to those scripts. The scripts take a fixed set of arguments (start, stop, restart, status) and do their work. The system runs them in a defined order, with each script blocking the next one from starting until it finishes.

Systemd is the modern approach. Services are unit files in /etc/systemd/system/ or /lib/systemd/system/. They are declarative: you describe what you want, and systemd figures out how to get there. Dependencies, ordering, restarts, environment variables, resource limits: all configured in the unit file. Systemd tracks the state of every service and can show you what is running, what failed, and why.

Most modern distributions (Ubuntu 16.04 and later, Debian 8 and later, CentOS 7 and later, Fedora 15 and later) use systemd by default. Some, like Devuan, antiX, and a handful of Alpine configurations, still use SysV. Docker containers usually use a minimal init that does not run any services, but you can install one.

The two systems can coexist. Systemd can run SysV scripts through a compatibility shim. The reverse is harder, but possible. If you write a service, knowing which system will run it matters.

The systemd unit file

A systemd unit file is the modern way to define a service. It is a plain text file with an .service extension. Here is a minimal example:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
Restart=on-failure
User=myapp
WorkingDirectory=/var/lib/myapp

[Install]
WantedBy=multi-user.target

Description is what shows up in systemctl status. After=network.target means “start this after the network is up.” It does not mean the network is required; for that, you use Requires= or Wants=.

The [Service] section is the actual work. Type=simple is the most common. It means systemd runs the command and considers the service started as soon as the process is running. Other types (forking, oneshot, notify) handle different cases. ExecStart is the command. Restart=on-failure tells systemd to restart the service if it crashes. User and WorkingDirectory are self-explanatory.

Inside [Install], you describe how the service gets enabled. WantedBy=multi-user.target creates a symlink in /etc/systemd/system/multi-user.target.wants/ when you run systemctl enable. That symlink is what causes the service to start at boot.

Once the file is in place, you reload systemd and start the service:

sudo systemctl daemon-reload
sudo systemctl start myapp
sudo systemctl enable myapp

daemon-reload is important after editing a unit file. Systemd caches the unit definitions, and without a reload, your changes will not take effect.

The SysV init script

A SysV init script is a shell script that responds to a fixed set of commands. The skeleton looks like this:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          myapp
# Required-Start:    $network $remote_fs
# Required-Stop:     $network $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

case "$1" in
  start)
    echo "Starting myapp"
    /usr/local/bin/myapp
    ;;
  stop)
    echo "Stopping myapp"
    pkill -f myapp
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  status)
    if pgrep -f myapp > /dev/null; then
      echo "myapp is running"
    else
      echo "myapp is not running"
      exit 1
    fi
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

The INIT INFO block is metadata. It tells the system what the script provides, what it depends on, and what runlevels (numbered states the system goes through during boot and shutdown) it should run in. Modern systems read this block to integrate the script with the rest of the init system.

The case statement is the contract. The script has to handle start, stop, restart, and status at minimum. The script is run as root (or with appropriate sudo), so it can do anything, but it should not assume interactive input. Services start at boot, before any user is logged in.

To install the script, copy it to /etc/init.d/, make it executable, and use the distribution’s tool to enable it:

sudo cp myapp /etc/init.d/
sudo chmod 755 /etc/init.d/myapp
sudo update-rc.d myapp defaults

On Red Hat-derived systems, the equivalent is chkconfig myapp on (older) or systemctl enable myapp (newer, which works because of the compatibility shim).

The commands you will use

For systemd, the day-to-day commands are a small set:

• systemctl start <service>: start the service now
• systemctl stop <service>: stop the service now
• systemctl restart <service>: stop and start the service
• systemctl reload <service>: ask the service to reload its config (only works if the service supports it)
• systemctl status <service>: show the current state and recent logs
• systemctl enable <service>: start the service at boot
• systemctl disable <service>: stop the service at boot
• systemctl list-units --type=service: list all loaded services
• journalctl -u <service>: show the logs for a service

For SysV, the commands are similar but use the service wrapper:

• service <name> start
• service <name> stop
• service <name> restart
• service <name> status

The service wrapper handles path differences between distributions and falls back to running the init script directly.

Debugging the common failures

A misconfigured unit file is the most common service failure. The second most common is a service that crashes immediately on startup. The third is a service that starts but is not reachable. Each has a different debugging approach.

When the unit file is the issue, systemctl status <service> will show you the error. Common mistakes include typos in ExecStart, missing executable permissions, and dependencies that do not exist. Running systemd-analyze verify <unit> performs a static check on the unit file and points out problems before the service is even started.

A service that crashes on startup usually leaves traces in journalctl -u <service> -n 100, which shows the last 100 log lines. If the service is writing to its own log file, check that file. The most common cause is a missing dependency: a database the service expects to be running, a configuration file that is not in the expected path, or a port that is already in use.

When the service is running but not reachable, the problem is usually network-related. Check that the service is listening on the expected port with ss -tlnp. Check that the firewall allows the connection with iptables -L or ufw status. Check that the service is binding to the right interface: binding to 127.0.0.1 means it is only reachable locally.

Logs: where to look

Systemd collects logs in its journal, accessed through journalctl. The journal is a binary log format that indexes by time, service, and priority. To see logs from a specific service: journalctl -u <service>. To follow new logs: journalctl -u <service> -f. To see logs from a specific time range: journalctl -u <service> --since "1 hour ago".

For SysV services, logs go wherever the service writes them. Most write to /var/log/. The exact location depends on the service. Nginx uses /var/log/nginx/. PostgreSQL uses /var/log/postgresql/. Custom services usually write to /var/log/<servicename>/ or to syslog.

Syslog itself is its own subsystem. On modern systems, syslog is often a thin shim over the journal. The traditional syslog config in /etc/rsyslog.conf controls where different message priorities go. For most purposes, the journal is the place to look.

Trade-offs

Both systems have trade-offs. SysV is simple and predictable. You can read the script and understand exactly what happens. Systemd is more powerful but more complex. The unit file is declarative, but debugging a systemd dependency chain can be harder than debugging a shell script.

• SysV init is portable. The same script runs on most Linux distributions. Systemd unit files have distribution-specific extensions and conventions.
• Systemd handles dependencies, restarts, resource limits, and logging in one place. SysV handles each of these in a separate tool.
• Systemd’s parallel startup is faster on modern hardware. SysV’s serial startup is slower but easier to reason about.
• Systemd has been controversial for years, with critics arguing it does too much. The debate is not going to settle soon.

For new services, systemd is the default on every major distribution. For legacy services, SysV still works. The two can coexist, and the compatibility shims are good enough that most users do not need to know which is which.

When to learn which

Learn systemd first. It is what your distribution is using, what your container is running, and what your cloud instance is booting. The unit file format is a small investment that pays off across the entire Linux landscape.

Learn SysV second, and only as much as you need. The basic structure of an init script is useful when you are debugging an old system or maintaining a script that has to run on both. Beyond that, the time is better spent on systemd, on the journal, and on the modern Linux service stack.