Debugging Bash Scripts (set -x)

Every script eventually does something you didn’t expect: a variable is empty when it shouldn’t be, a loop runs the wrong number of times, or a command silently fails and the rest of the script keeps going anyway. Bash gives you a built-in tracer, set -x, that prints every command it runs along with its expanded arguments before executing it. Combined with a few related options and habits, it turns “why is this broken” into a fast, systematic process instead of guesswork.

Overview / How it works

Bash normally runs a script silently: it reads a line, expands variables and substitutions, and executes the result, printing only what the commands themselves print. set -x (short for xtrace) turns on a tracing mode where, before executing each command, Bash prints the command after all expansions (variables, command substitution, globbing) have been applied, prefixed with +. This is different from just reading the source: you see what the shell actually resolved $count or $(date) to at that moment, which is exactly the information you need when a bug depends on runtime data.

Xtrace is a shell option, not a separate program. It can be turned on for an entire script invocation from the command line, toggled on and off around a specific block of code with set -x / set +x, or enabled automatically by putting set -x near the top of the script. Internally, Bash tracks this as one of several shell options (alongside -e for exit-on-error and -u for treating unset variables as errors) that can be combined, e.g. set -euxo pipefail for a script that stops on the first error, treats unset variables as fatal, and traces everything.

Each traced line is prefixed by the value of the PS4 variable, which defaults to + . You can customize PS4 to include the script name, line number, or function name, which becomes essential once a script is more than a few dozen lines, because plain + markers don’t tell you where in the file a given traced command lives.

It helps to know that xtrace only shows you commands as the shell parses and expands them — it does not show you what a command’s own internal logic does, and it does not by itself tell you whether a command succeeded or failed. For that you still need $? (the previous command’s exit status) or set -e to stop execution on failure. Debugging effectively usually means combining xtrace (to see what ran) with exit-status checking (to see what failed).

Syntax

set -x        # turn xtrace on for the rest of the script/session
set +x        # turn xtrace off
bash -x script.sh   # run a whole script with xtrace on, without editing it
PS4='+ ${BASH_SOURCE}:${LINENO}: '   # customize the trace prefix

Form	What it does
set -x	Enable xtrace from this point forward in the current shell/script.
set +x	Disable xtrace (the + here means “turn off”, opposite of -).
bash -x script.sh	Run script.sh with tracing on for its entire execution, without adding set -x to the file.
set -v	Verbose mode: prints each raw source line before expansion (as written in the file), unlike -x which prints it after expansion.
PS4	The prompt string printed before each traced line; defaults to + , customizable with variables like $LINENO and $BASH_SOURCE.
set -e	Exit immediately if any command exits non-zero (pairs well with -x to see exactly which traced command killed the script).
set -u	Treat references to unset variables as an error instead of silently expanding to an empty string.

Examples

Example 1: tracing a loop with set -x

#!/usr/bin/env bash
set -x

count=5
total=0
for i in $(seq 1 "$count"); do
  total=$((total + i))
done
echo "Total: $total"

set +x

+ count=5
+ total=0
++ seq 1 5
+ for i in $(seq 1 "$count")
+ total=1
+ for i in $(seq 1 "$count")
+ total=3
+ for i in $(seq 1 "$count")
+ total=6
+ for i in $(seq 1 "$count")
+ total=10
+ for i in $(seq 1 "$count")
+ total=15
+ echo 'Total: 15'
Total: 15
+ set +x

Notice two things: command substitutions like $(seq 1 "$count") get their own extra-indented ++ trace line because Bash runs them in a subshell first, and every loop iteration is shown separately with the already-expanded values of total. This is the core value of set -x — you don’t have to guess what $count expanded to, you can read it directly.

Example 2: tracing without editing the script

bash -x backup.sh /var/log/app.log

+ SRC=/var/log/app.log
+ DEST=/var/backups/app.log.bak
+ cp /var/log/app.log /var/backups/app.log.bak
+ echo 'Backed up /var/log/app.log'
Backed up /var/log/app.log

Running bash -x script.sh traces the whole script for that one invocation without permanently adding set -x to the file. This is the safest way to debug a script you don’t own or don’t want to modify — nothing about the script itself changes.

Example 3: customizing PS4 to show file and line number

#!/usr/bin/env bash
PS4='+ ${BASH_SOURCE}:${LINENO}: '
set -x

log_file="/var/log/app.log"
if [[ ! -f "$log_file" ]]; then
  echo "Missing: $log_file" >&2
  exit 1
fi
echo "Found: $log_file"

+ deploy.sh:5: log_file=/var/log/app.log
+ deploy.sh:6: [[ ! -f /var/log/app.log ]]
+ deploy.sh:9: echo 'Found: /var/log/app.log'
Found: /var/log/app.log

With a custom PS4, every traced line tells you exactly which file and line number produced it. In a script with functions, sourced files, or hundreds of lines, this is the difference between finding a bug in seconds and scrolling through undifferentiated + lines trying to count which one is which.

How it works step by step

When Bash executes a line with xtrace enabled, it follows this order for each command: (1) read the raw line from the script, (2) perform expansions — variable substitution ($var), command substitution ($(...)), arithmetic expansion ($((...))), and globbing (*.log), (3) print the expanded form to standard error, prefixed with PS4, and (4) execute the expanded command. Because tracing happens after expansion, you see reality — the actual file names, numbers, and command-line arguments involved — not the literal source text (that’s what set -v is for instead). Because the trace is written to standard error (not standard output), it won’t get mixed into a script’s normal output if that output is redirected or piped, but it will still show up on your terminal unless you also redirect file descriptor 2.

Common Mistakes

Mistake 1: leaving set -x on and shipping it

#!/usr/bin/env bash
set -x
db_password="$1"
mysql -u admin -p"$db_password" -e "SELECT 1;"

Because set -x prints every command after expansion, it will print the actual value of $db_password into the trace output — which likely ends up in a log file. Turn tracing off before anything sensitive runs, or better, never trace scripts that handle secrets in production; keep set -x as a debugging tool you enable on demand with bash -x, not a permanent line in a deployed script.

#!/usr/bin/env bash
db_password="$1"
mysql -u admin -p"$db_password" -e "SELECT 1;"
# run with: bash -x script.sh '' only while debugging, never in production

Mistake 2: reading trace output but ignoring exit status

set -x
cp /var/log/app.log /var/backups/
rm /var/log/app.log
echo "Rotated log"

Xtrace shows you that cp and rm ran, but not whether cp actually succeeded. If the backup directory doesn’t exist, cp fails, yet the script happily deletes the original log anyway and prints “Rotated log” as if nothing went wrong. Pair set -x with set -e (or an explicit check) so a failed step stops the script instead of silently continuing.

#!/usr/bin/env bash
set -euxo pipefail
cp /var/log/app.log /var/backups/
rm /var/log/app.log
echo "Rotated log"

Mistake 3: unquoted variables making trace output misleading

set -x
file="my report.log"
cat $file

The trace line for this shows + cat 'my' 'report.log' style word-splitting problems — cat receives two arguments instead of one, and fails with “No such file” for both. The trace output is telling you the truth about the bug (word-splitting), but only if you quote variables in general does the rest of the script behave predictably; unquoted expansion should be treated as the mistake it is, not routine style.

set -x
file="my report.log"
cat "$file"

Best Practices

• Prefer bash -x script.sh over hardcoding set -x in the file when you’re debugging someone else’s script or a script you don’t want to permanently modify.
• When you do add set -x inside a script, scope it tightly with a matching set +x around just the suspicious block, rather than tracing the whole script.
• Customize PS4 to include ${BASH_SOURCE} and ${LINENO} so trace output is traceable back to exact source lines, especially in scripts with functions or sourced files.
• Combine set -x with set -euo pipefail while debugging so the script stops at the first real failure instead of scrolling past it.
• Never leave set -x enabled in a production script that touches passwords, API keys, or tokens — expanded values land directly in the trace output.
• Redirect trace output separately from program output when needed with bash -x script.sh 2>trace.log, since xtrace writes to standard error.
• Use set -v instead of, or alongside, -x when you specifically need to see the raw source line as written, not just the expanded command.

Practice Exercises

• Write a script disk_check.sh that computes free disk space with df and compares it to a threshold using arithmetic. Introduce a deliberate bug (e.g. compare the wrong variable), then run it with bash -x disk_check.sh and use the trace output to find and fix the bug.
• Take any script you’ve already written for this course and add a custom PS4 with ${LINENO}, then run it with set -x enabled for just one function using set -x / set +x around the function call.
• Write a two-command script where the first command can plausibly fail (e.g. cp to a nonexistent directory). Run it once with only set -x and once with set -euxo pipefail, and compare how differently the script behaves after the failure.

Summary

• set -x (xtrace) prints each command, after expansion, prefixed by PS4, before executing it.
• Use set -x / set +x to scope tracing to a block, or bash -x script.sh to trace a whole run without editing the file.
• Customize PS4 with ${BASH_SOURCE} and ${LINENO} to make trace output point straight to the source line.
• set -x shows what ran, not whether it succeeded — combine it with set -e and checking $? to catch silent failures.
• Never trace a script that handles secrets in production; expanded variable values, including passwords, appear directly in the trace.
• Unquoted variables produce misleading, word-split trace output — always quote expansions ("$var") in real scripts.