Viewing Logs (journalctl, /var/log)

Every process on a Linux system quietly narrates what it’s doing: services start and stop, the kernel notices hardware, SSH records login attempts, and your own scripts can print status messages. All of that narration ends up in logs. Knowing where logs live and how to read them is one of the most important system administration skills, because when something breaks, the logs are almost always where you find out why. This lesson covers the two main ways to view logs on a modern Linux system: the journalctl command for the systemd journal, and the traditional plain-text files under /var/log.

Overview: How Logging Works on Linux

On most modern distributions (Ubuntu, Debian, Fedora, RHEL), logging is handled by two cooperating systems: systemd-journald and, often, rsyslog.

systemd-journald is a system service that starts very early in boot and immediately begins collecting log messages from three main sources: the kernel (via a kernel log ring buffer), any process that writes to the traditional syslog socket, and, most importantly, the standard output and standard error of every service that systemd itself starts. Because systemd launches almost every long-running daemon on the system (via unit files), it can capture a service’s stdout/stderr directly without that service needing to know anything about logging at all — this is one of journald’s biggest advantages over the old syslog-only world.

journald stores what it collects in a binary, indexed format (not plain text) under /var/log/journal/ (persistent, survives reboots) or /run/log/journal/ (volatile, wiped at reboot, used when persistent storage isn’t configured). Because it’s indexed, you can query it efficiently by time range, by service (“unit”), by priority level, or by boot session — something that’s slow and awkward to do with a giant plain-text file. You never read this binary format directly; you always go through the journalctl command, which parses and formats it for you.

rsyslog (or the newer syslog-ng) is the traditional Unix syslog daemon. On many distributions it still runs alongside journald, receiving a forwarded copy of journal messages and writing them out as classic plain-text files in /var/log — files like /var/log/syslog (Debian/Ubuntu, general system messages), /var/log/auth.log (Debian/Ubuntu authentication and sudo events), or on RHEL/Fedora /var/log/messages and /var/log/secure respectively. Some applications that aren’t managed by systemd, or that predate journald, write their own log files directly into /var/log too — for example /var/log/nginx/access.log or /var/log/apache2/error.log.

So in practice you have two toolsets for two kinds of logs: journalctl for anything systemd manages (which today is most services), and ordinary text tools (cat, less, tail, grep) for the flat files in /var/log. Knowing both is essential, because not every log ends up in the journal, and not every log ends up in a text file.

Syntax

journalctl [options]

With no arguments, journalctl shows the entire journal, oldest entries first, piped through a pager (usually less). The most useful options:

Option Meaning
-u <unit> Show only messages from a specific systemd unit (service), e.g. -u nginx.service
-f Follow mode — keep the terminal open and print new entries as they arrive, like tail -f
-e Jump straight to the end of the output (the most recent entries)
-n <N> Show only the last N entries (default 10 if used alone)
-r Reverse order: newest entries first
-b Show only messages from the current boot; -b -1 means the previous boot
-k Show only kernel messages (equivalent to the old dmesg)
-p <level> Filter by minimum priority: emerg, alert, crit, err, warning, notice, info, debug
--since, --until Filter by time, accepts values like "2026-08-01 09:00:00" or relative phrases like "1 hour ago"
-o <format> Output format, e.g. json-pretty for machine-readable output
--no-pager Print directly to the terminal instead of opening a pager (useful in scripts)
-x Add explanatory help text to messages where systemd has extra context available
--disk-usage Show how much disk space the journal is currently using
--vacuum-size, --vacuum-time Shrink the journal to a target size or delete entries older than a given time

Note that reading the full journal (all users’ messages, kernel messages, etc.) typically requires membership in the systemd-journal group or running as root — in practice, that means using sudo.

Examples

Example 1: Viewing logs for one service

Suppose an nginx web server won’t start and you want to see what happened during the current boot.

journalctl -u nginx.service -b

Output:

Aug 04 09:12:01 web01 systemd[1]: Starting nginx.service - A high performance web server...
Aug 04 09:12:01 web01 nginx[2211]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Aug 04 09:12:01 web01 systemd[1]: nginx.service: Control process exited, code=exited status=1
Aug 04 09:12:01 web01 systemd[1]: nginx.service: Failed with result 'exit-code'.
Aug 04 09:12:01 web01 systemd[1]: Failed to start nginx.service - A high performance web server.

The -u flag scopes the output to just that unit, and -b limits it to the current boot so you’re not scrolling through weeks of history. Here the log immediately reveals the real problem: something else is already bound to port 80.

Example 2: Following logs live

To watch a service’s logs in real time while you reproduce a bug — similar to tail -f on a text file — use -f:

journalctl -u nginx.service -f

Output (streams continuously until you press Ctrl+C):

Aug 04 10:03:44 web01 nginx[3350]: 203.0.113.7 - - [04/Aug/2026:10:03:44 +0000] "GET /health HTTP/1.1" 200 2
Aug 04 10:03:59 web01 nginx[3350]: 203.0.113.9 - - [04/Aug/2026:10:03:59 +0000] "GET /api/status HTTP/1.1" 200 15

This is the same technique you’d use with tail -f, but it works even for services that only log through journald and never write a plain-text file.

Example 3: Filtering by time and severity

To find only errors from the last hour, across all services:

journalctl --since "1 hour ago" -p err --no-pager

Output:

Aug 04 09:41:12 web01 kernel: I/O error, dev sda, sector 20971520
Aug 04 09:55:03 web01 cron[1882]: (root) MAIL (mailed 2 bytes of output but got status 0x004b)

-p err shows this level and everything more severe (err, crit, alert, emerg), filtering out the informational noise. Combined with --since, this is exactly the kind of query you’d run first thing when investigating an incident.

Example 4: Reading plain-text logs in /var/log

Not everything lives in the journal. To check for failed SSH login attempts, which Debian/Ubuntu records in /var/log/auth.log:

grep "Failed password" /var/log/auth.log

Output:

Aug 04 03:12:09 web01 sshd[9931]: Failed password for root from 198.51.100.23 port 51022 ssh2
Aug 04 03:12:11 web01 sshd[9931]: Failed password for root from 198.51.100.23 port 51030 ssh2

These plain-text files respond to the normal text tools: cat for short files, less for paging through large ones, tail -f to follow them live, and grep to search them. On RHEL/Fedora the equivalent file is /var/log/secure.

How It Works, Step by Step

    Walking through what happens when a service like nginx writes a log message:

    • The nginx process writes a line to its standard error stream, exactly like calling echo to the terminal — nginx itself doesn’t know or care that systemd is involved.
    • Because systemd started nginx as a unit, it had already connected nginx’s stdout/stderr file descriptors to a socket that journald listens on, rather than to a terminal.
    • systemd-journald receives the line, attaches metadata to it automatically: a timestamp, the unit name, the process ID, the boot ID, the priority level, and more.
    • journald appends this structured entry to its binary journal file under /var/log/journal/ (if persistent storage is enabled) or /run/log/journal/ (volatile only).
    • If rsyslog is also running, journald forwards a copy of the message to it, and rsyslog writes a corresponding line into a flat file such as /var/log/syslog based on its own rules.
    • When you run journalctl -u nginx.service, journalctl opens the indexed binary journal, applies your filters (unit, time range, priority), and formats matching entries as readable text — it never scans a giant flat file line by line the way grep does.

    This explains why journalctl queries stay fast even on a system with months of logs, while grep-ing a multi-gigabyte /var/log/syslog can be noticeably slower — one is an indexed query, the other is a linear scan.

    Common Mistakes

    Mistake 1: Running journalctl without sudo and wondering why logs are missing

    Unprivileged users typically only see their own messages, not the full system journal:

    journalctl -u ssh.service

    Output:

    Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No entries

    The fix is to run it with elevated privileges, which is the normal, logged way to read system-wide logs:

    sudo journalctl -u ssh.service

    Mistake 2: Using > instead of >> when saving log snapshots

    Redirecting journalctl output to a file to keep a record seems reasonable, but > truncates the file every time:

    journalctl -u nginx.service --no-pager > nginx-issues.log
# ...later that day, capturing more output...
journalctl -u nginx.service --since today --no-pager > nginx-issues.log

    The second command silently erases the first capture, because > always overwrites the file from scratch. Use >> to append instead:

    journalctl -u nginx.service --no-pager >> nginx-issues.log

    Mistake 3: Unquoted variables when scripting log lookups

    It’s tempting to loop over a list of services using a plain string, but without quoting, word-splitting can silently change what gets passed to -u:

    SERVICES="nginx.service postgresql.service"
for svc in $SERVICES; do
  journalctl -u $svc --no-pager
done

    This particular case happens to work because the values contain no spaces or glob characters, but it’s fragile: as soon as a unit name or variable ever contains a space, wildcard character, or is unset, the unquoted expansion breaks unpredictably. Use an array and quote every expansion:

    #!/usr/bin/env bash
set -euo pipefail

SERVICES=("nginx.service" "postgresql.service")

for svc in "${SERVICES[@]}"; do
  journalctl -u "$svc" --no-pager -n 20
done

    Best Practices

    • Start incident investigation with -p err and a --since window rather than scrolling through the entire journal.
    • Prefer -u <unit> over grepping flat files whenever the service in question runs under systemd — it’s faster and gives you clean, per-service output.
    • Check journal disk usage periodically with journalctl --disk-usage; an unbounded journal can fill up /var/log and, on some setups, the root filesystem.
    • Set retention limits (SystemMaxUse= in /etc/systemd/journald.conf, or run journalctl --vacuum-time=2weeks) instead of letting logs grow forever.
    • Always quote variables and command substitutions in log-processing scripts — log lines routinely contain spaces, quotes, and special characters that will break unquoted expansions.
    • Use --no-pager when calling journalctl from a script or when piping its output into another command, so it doesn’t wait for a pager to be closed.
    • Remember that not everything is in the journal: applications that log directly to their own files (many web servers, databases) still need to be checked under /var/log/<app>/.
    • Use sudo for full log access rather than logging in as root, so the access itself is attributable and audited.

    Practice Exercises

    • Exercise 1: Find every log entry generated during the current boot for the cron service, newest first. Hint: combine -u, -b, and -r.
    • Exercise 2: Your disk is filling up. Check how much space the systemd journal is using, then shrink it down to at most 100 MB without deleting the whole thing.
    • Exercise 3: Write a short Bash script that takes a service name as its first argument ($1), prints the last 20 journal entries for that service, and exits with an error message if no argument was given. Make sure every variable expansion is quoted and the script uses set -euo pipefail.

    Summary

    • systemd-journald collects logs from the kernel, syslog, and every systemd-managed service into an indexed binary journal, readable only through journalctl.
    • rsyslog (where present) writes a parallel copy of many messages into plain-text files under /var/log, such as /var/log/syslog and /var/log/auth.log on Debian/Ubuntu, or /var/log/messages and /var/log/secure on RHEL/Fedora.
    • Key journalctl flags: -u (unit), -f (follow), -b (this boot), -p (priority), --since/--until (time range).
    • Reading the full journal generally requires sudo or membership in the systemd-journal group.
    • Use >>, not >, when appending captured log output to a file, and always quote variables in log-processing scripts.
    • Monitor and cap journal disk usage with --disk-usage and --vacuum-size/--vacuum-time so logs don’t silently consume all your disk space.