systemd and Services (systemctl)

Every running Linux system has one process that starts first and outlives all others: PID 1. On virtually every modern distribution — Ubuntu, Debian, Fedora, RHEL — that process is systemd, and systemctl is the command you use to talk to it. Whether you need a web server to start automatically on boot, a background worker to restart itself after a crash, or you just want to know why an application silently stopped responding, you are working with systemd. This lesson covers how systemd actually works under the hood and how to use systemctl to control it confidently.

Overview: How systemd Works

When the kernel finishes booting, it mounts the root filesystem and hands control to a single userspace program: the init system. For decades that was SysV init, a set of shell scripts run in numbered sequence. Modern distributions replaced it with systemd, a much more capable init system and service manager. systemd becomes PID 1, meaning it is the parent (directly or indirectly) of every other process on the system, and it is responsible for reaping orphaned processes, starting services in the correct order, and keeping the system in a known state.

systemd organizes everything it manages into units. A unit is a configuration file describing something systemd should start, stop, or monitor. The most common unit type is the service unit (.service), which represents a long-running or one-shot program, but there are others: .socket units listen on a network or Unix socket and start a service on demand, .mount units manage filesystem mounts, .timer units trigger other units on a schedule (a modern alternative to cron), and .target units group other units together as a synchronization point — for example multi-user.target is roughly equivalent to the old SysV “runlevel 3” (a working non-graphical multi-user system), and graphical.target adds a display manager on top of it.

Unit files live in three main places, searched in order of precedence: /etc/systemd/system/ (local administrator overrides and custom units — highest priority), /run/systemd/system/ (runtime-generated units), and /usr/lib/systemd/system/ (units installed by packages via apt or dnf — lowest priority, and overwritten on package upgrades). This is why you should never hand-edit a unit file in /usr/lib/systemd/system/: the next package update will silently discard your changes.

A service unit file has three common sections. [Unit] holds metadata and ordering: Description, and dependency directives like After= (start after another unit, but don’t require it) and Requires= (a hard dependency — if it fails, this unit fails too). [Service] describes how to run the program: Type= (e.g. simple for a process that stays in the foreground, or oneshot for a script that runs and exits), ExecStart= (the command to run), Restart= (what to do if it dies), and User= (which account to run as instead of root). [Install] describes what happens when the unit is enabled, typically WantedBy=multi-user.target, meaning “when the system reaches multi-user.target during boot, start me too.”

Under the hood, systemd tracks every service’s processes using a Linux kernel feature called cgroups (control groups). This lets systemd reliably know every process that belongs to a service — including ones the service itself forked — so that systemctl stop can clean up the entire process tree, not just the one PID it originally started. systemd also builds a dependency graph from every unit’s After=/Before=/Requires=/Wants= directives and starts independent units in parallel, which is a major reason systemd boots faster than the old sequential SysV scripts.

Syntax

The general form of the command is:

systemctl <command> [unit-name]
Command What it does
status Shows whether a unit is running, its recent log lines, and its main PID.
start / stop Starts or stops a unit right now (does not affect boot behavior).
restart / reload restart stops then starts; reload asks the running process to re-read its config without restarting.
enable / disable Creates or removes the symlink that makes a unit start automatically at boot (does not start/stop it now).
enable --now Enables and starts the unit in one command.
is-active / is-enabled Prints active/inactive or enabled/disabled and sets the exit code accordingly — useful in scripts.
mask / unmask Links the unit to /dev/null so it cannot be started at all, even manually, until unmasked.
list-units --type=service Lists currently loaded service units and their state.
list-units --failed Lists units that failed to start — the first place to look after a bad boot.
daemon-reload Tells systemd to re-read unit files from disk after you edit or add one.
cat Prints the effective contents of a unit file (including overrides).
edit Opens an editor to create a safe override snippet for a unit without touching the vendor file.

Examples

Example 1: Checking and controlling an installed service

Suppose nginx is installed on a server and you want to check whether it’s running, then make sure it starts on every boot:

sudo systemctl status nginx.service

Output:

● nginx.service - A high performance web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

The service is loaded (systemd knows about it) but inactive and disabled (not running, and won’t start at boot). Enable it and start it in one step:

sudo systemctl enable --now nginx.service

Output:

Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.

The symlink is the actual mechanism behind “enable”: systemd’s boot sequence walks multi-user.target.wants/ and starts everything linked there. Verify both states independently:

systemctl is-enabled nginx.service
systemctl is-active nginx.service

Output:

enabled
active

Example 2: A custom service for a script

Say you have a backup report script and want systemd to run it as a proper service, as a dedicated backupuser instead of root. First, install the script:

#!/usr/bin/env bash
set -euo pipefail

LOG_DIR="/var/log/backup-report"
BACKUP_DIR="/var/backups/app"
mkdir -p "$LOG_DIR"

if [[ -d "$BACKUP_DIR" ]]; then
    du -sh "$BACKUP_DIR" >> "$LOG_DIR/report.log"
    echo "$(date '+%Y-%m-%d %H:%M:%S') backup report generated" >> "$LOG_DIR/report.log"
else
    echo "$(date '+%Y-%m-%d %H:%M:%S') backup directory missing: $BACKUP_DIR" >&2
    exit 1
fi
sudo cp backup-report.sh /usr/local/bin/backup-report.sh
sudo chmod +x /usr/local/bin/backup-report.sh

Then create the unit file at /etc/systemd/system/backup-report.service with a text editor (this is a systemd unit file, written in INI-style config syntax — not a Bash script):

[Unit]
Description=Generate nightly backup report
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-report.sh
User=backupuser

[Install]
WantedBy=multi-user.target

systemd does not notice new or changed unit files on its own — you must tell it to re-read them, then enable and run it:

sudo systemctl daemon-reload
sudo systemctl enable --now backup-report.service

Because Type=oneshot means the script is expected to exit, systemctl status will show it as active (exited) once it finishes successfully rather than staying “active (running)” like nginx does.

Example 3: Reading logs and finding failures

systemd captures the stdout/stderr of every service it manages into the journal, viewable with journalctl, filtered by unit:

journalctl -u backup-report.service --since today

Output:

Aug 04 02:00:01 web01 systemd[1]: Starting Generate nightly backup report...
Aug 04 02:00:01 web01 systemd[1]: backup-report.service: Deactivated successfully.
Aug 04 02:00:01 web01 systemd[1]: Finished Generate nightly backup report.

If a service is crash-looping, systemctl list-units --failed combined with journalctl -u <unit> -n 50 (last 50 lines) is the standard first move for diagnosing it.

How systemctl Works Step by Step

When you run sudo systemctl start backup-report.service, here is what actually happens:

  1. systemctl is a client program — it does not start anything itself. It sends a request over a D-Bus socket to the systemd process (PID 1).
  2. systemd looks up the unit in its internal table, built from the last daemon-reload, and checks the unit’s state machine (currently inactive).
  3. systemd resolves the unit’s dependencies (After=, Requires=) and starts any that aren’t already running first.
  4. systemd forks a child process, sets up its cgroup, applies the User=/environment settings, and execs the command in ExecStart=.
  5. The unit transitions through activating to active (or active (exited) for a oneshot that completed with exit code 0).
  6. If the process later exits non-zero and Restart=on-failure is set, systemd notices via the cgroup, waits RestartSec=, and repeats the start sequence automatically.

At boot, the same mechanism runs at scale: systemd starts at default.target (usually a symlink to graphical.target or multi-user.target), walks the full dependency graph, and starts every reachable unit, launching independent branches of the graph in parallel rather than one at a time.

Common Mistakes

Mistake 1: Editing the vendor unit file directly

Editing files under /usr/lib/systemd/system/ feels natural but is fragile:

sudo nano /usr/lib/systemd/system/nginx.service

The next apt upgrade of the nginx package will overwrite this file and silently discard your change. Use an override instead, which systemd merges on top of the vendor file:

sudo systemctl edit nginx.service

Mistake 2: Forgetting daemon-reload after changing a unit file

systemd caches parsed unit files in memory. If you hand-edit a file and immediately restart the service, you’re still running the old configuration:

sudo nano /etc/systemd/system/backup-report.service
sudo systemctl restart backup-report.service

The fix is to reload the unit database before restarting:

sudo nano /etc/systemd/system/backup-report.service
sudo systemctl daemon-reload
sudo systemctl restart backup-report.service

Mistake 3: Confusing enable with start

These control two independent things — “runs now” and “runs on next boot” — and mixing them up is one of the most common systemd mistakes:

sudo systemctl enable nginx.service

This creates the boot-time symlink but does not start nginx right now — a reader who expects the site to be live immediately will be confused when it isn’t. Use --now when you want both:

sudo systemctl enable --now nginx.service

Symmetrically, systemctl stop only stops a service for the current session; if it’s still enabled, it comes back on the next reboot. To stop it permanently, also run systemctl disable.

Best Practices

  • Use systemctl edit <unit> for customizing an existing (vendor-shipped) service rather than editing its file directly — it creates a safe override in /etc/systemd/system/<unit>.d/.
  • Always run systemctl daemon-reload immediately after creating or editing any unit file.
  • Use systemctl cat <unit> to see the final, merged configuration systemd is actually using, including overrides.
  • Set Restart=on-failure (not blindly Restart=always) on services that should recover from crashes, and pair it with RestartSec= to avoid rapid restart loops.
  • Check systemctl list-units --failed after any deployment or reboot to catch problems early.
  • Prefer a dedicated, non-root User= for custom services instead of running everything as root.
  • Use journalctl -u <unit> as your first debugging step for a misbehaving service before digging through application-specific log files.
  • Use .timer units instead of cron when the job is closely tied to a systemd service, since timers integrate with journal logging and dependency ordering.

Practice Exercises

  1. Write a small Bash script that appends the current disk usage of / to a log file, install it under /usr/local/bin/, and create a systemd oneshot service that runs it. Enable and start it, then confirm it ran successfully using systemctl status and journalctl -u.
  2. Take the service from the previous exercise, stop it, then intentionally break it by pointing ExecStart= at a script path that doesn’t exist. Reload, restart, and use systemctl status plus journalctl to identify the failure reason from the output.
  3. Pick any running service on your system (e.g. ssh.service) and use systemctl edit to add an override that sets Restart=on-failure and RestartSec=5, without touching the original unit file. Confirm your override took effect with systemctl cat.

Summary

  • systemd is PID 1 on most modern Linux distributions and manages the whole system as a graph of units, most commonly .service units.
  • Unit files in /etc/systemd/system/ override those in /usr/lib/systemd/system/, which are owned by installed packages and get overwritten on upgrade.
  • enable controls whether a unit starts at boot; start controls whether it’s running right now — they are independent, and --now combines them.
  • After creating or editing any unit file, run systemctl daemon-reload before the change takes effect.
  • journalctl -u <unit> is the standard way to see a service’s logs and diagnose failures.
  • systemd tracks a service’s full process tree via cgroups, which is how stop reliably kills everything a service spawned.