Pipes and Chaining Commands

A pipe lets you take the output of one command and feed it directly into another command as input, without ever writing an intermediate file to disk. Chaining takes this further: you can link several commands together based on whether each one succeeds or fails, building small pipelines of logic right on the command line. Together, pipes and chaining are what let Linux’s small, single-purpose tools (grep, sort, uniq, wc, and dozens more) combine into powerful one-line programs. This lesson covers how pipes actually work at the process level, how to build multi-stage pipelines, and how to chain commands together with &&, ||, and ;.

Overview / How it works

Every running process on Linux starts with three open file descriptors: 0 (standard input, stdin), 1 (standard output, stdout), and 2 (standard error, stderr). By default, a command reads from the keyboard (stdin) and writes both its normal output and its error messages to the terminal (stdout and stderr). Redirection, which you saw in earlier lessons, points those file descriptors at files instead. A pipe does something related but different: it points one process’s stdout directly at another process’s stdin, with no file involved at all.

When Bash sees command1 | command2, it does the following, roughly: it asks the kernel to create a pipe (a small, in-memory, one-directional buffer with a read end and a write end), then it forks two child processes. The first child has its stdout (file descriptor 1) redirected to the pipe’s write end, and the second child has its stdin (file descriptor 0) redirected to the pipe’s read end. Both processes are started at the same time and run concurrently — the kernel schedules them independently. As command1 produces data, the kernel holds it in the pipe’s buffer; as soon as any data is available, command2 can start reading it, even before command1 has finished. This is why a pipeline like tail -f app.log | grep ERROR can show matching lines in real time as they’re written, instead of waiting for the whole file.

Crucially, a pipe only connects standard output to standard input — it does not touch standard error by default. If a command in your pipeline writes error messages, those still go straight to the terminal (or wherever fd 2 currently points), bypassing the pipe entirely. This trips up a lot of people, and it’s covered in Common Mistakes below.

You can chain as many commands as you like: cmd1 | cmd2 | cmd3 | cmd4 creates a pipeline of four processes, each one’s stdout wired to the next one’s stdin. The shell manages all of this bookkeeping for you; you just write the pipe characters.

Chaining by exit status: &&, ||, and ;

Pipes connect data between commands. The &&, ||, and ; operators connect commands based on their exit status instead — the number (0–255) a process returns to the shell when it finishes, where 0 conventionally means success and anything nonzero means some kind of failure. cmd1 && cmd2 runs cmd2 only if cmd1 exits with status 0. cmd1 || cmd2 runs cmd2 only if cmd1 exits nonzero. cmd1 ; cmd2 just runs both in sequence regardless of what cmd1 did. These are how you write “do this, and only if it works, do that” directly on the command line or in a script, without a full if statement.

Syntax

command1 | command2 | command3

command1 && command2   # run command2 only if command1 succeeds (exit 0)
command1 || command2   # run command2 only if command1 fails (exit nonzero)
command1 ; command2    # run command2 regardless of command1's result
Operator Meaning
| Connects stdout of the left command to stdin of the right command
&& Run the next command only if the previous one exited 0 (success)
|| Run the next command only if the previous one exited nonzero (failure)
; Run commands one after another unconditionally
2>&1 Redirect stderr into wherever stdout currently points (often used right before a pipe)

Examples

Example 1: filtering process output. Suppose you want to check whether the nginx web server is running.

ps aux | grep nginx

Output:

root         1198  0.0  0.1  55432  3040 ?        Ss   08:01   0:00 nginx: master process
www-data     1234  0.0  0.3  55432  8120 ?        S    08:01   0:00 nginx: worker process
you          5567  0.0  0.0  17652  1088 pts/0    S+   09:14   0:00 grep --color=auto nginx

ps aux writes a full process listing to its stdout. That output becomes the stdin of grep, which only prints lines containing “nginx”, discarding everything else. Note that grep itself shows up in the results, since its own command line contains the word “nginx” — a common cosmetic annoyance. You can avoid it with a bracket trick like grep '[n]ginx', which matches the literal text but no longer matches the pattern of the grep command itself.

Example 2: a three-stage pipeline over a log file. Find the five IP addresses that show up most often in an nginx access log.

cut -d ' ' -f1 /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -5

Output:

    245 203.0.113.42
    198 198.51.100.7
     87 203.0.113.99
     42 192.0.2.15
     16 198.51.100.201

Each stage does one small job. cut -d ' ' -f1 pulls just the first space-separated field (the IP address) from every line. sort puts identical IPs next to each other, which uniq -c requires in order to count consecutive duplicates — it collapses runs of identical lines and prefixes each with a count. The second sort -rn then orders those counts numerically (-n) in reverse (-r), largest first, and head -5 keeps only the top five lines. Five simple tools, none of which knows anything about the others, combine into a real analysis.

Example 3: chaining on success and failure. Back up a directory and report clearly whether it worked.

mkdir -p /backups && tar -czf "/backups/app-$(date +%F).tar.gz" /var/www/app && echo "Backup complete" || echo "Backup FAILED" >&2

Output:

Backup complete

This runs mkdir -p /backups, and only if that succeeds does it run tar to create the archive; only if that succeeds does it print “Backup complete”. If any command in the && chain fails, the whole chain short-circuits and control falls through to the ||, printing “Backup FAILED” to stderr instead. $(date +%F) is a command substitution that forks a subshell to run date +%F and substitutes its output (like 2026-08-04) into the filename, and it’s quoted so the resulting path is treated as one argument even though it’s built from a shell expansion.

How it works step by step

Walking through cut -d ' ' -f1 /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -5:

  1. Bash parses the line and sees four pipe characters, meaning five commands to connect.
  2. The kernel creates four pipes (one between each adjacent pair of commands).
  3. Bash forks five child processes: cut, sort, uniq -c, sort -rn, and head -5. Each one gets its stdin and/or stdout wired to the appropriate pipe end instead of the terminal.
  4. All five processes start essentially simultaneously and run concurrently, each blocking on its pipe’s read end until data arrives, and blocking on its pipe’s write end if the next process hasn’t read fast enough yet (the kernel’s pipe buffer is finite).
  5. cut reads the log file directly (it isn’t piped in, it’s given as an argument) and streams IP addresses out as it reads. sort can’t produce any output until it has seen every line, since sorting requires the whole input; the same is true for the second sort -rn.
  6. Once every process in the pipeline has exited, the shell’s $? holds the exit status of the last command in the pipeline — head in this case — and prints the prompt back.

Common Mistakes

Mistake 1: the “useless use of cat.” A very common but wasteful pattern is piping a file’s contents into a command that could just read the file itself:

cat /var/log/app.log | grep "ERROR"

This spawns an extra process just to feed data to grep, which can read files directly. The corrected version:

grep "ERROR" /var/log/app.log

Mistake 2: assuming a pipe carries stderr too. Suppose you run a build script and pipe its output to tee to save a log while still watching it on screen:

./build.sh | tee build.log

If build.sh writes its error messages to stderr (fd 2), as well-behaved programs do, those lines never enter the pipe at all — they print straight to the terminal and never make it into build.log. To capture both streams, redirect stderr into stdout before the pipe:

./build.sh 2>&1 | tee build.log

Mistake 3: checking $? after a pipeline and getting the wrong command’s status. By default, $? after a pipeline reflects only the last command’s exit status, not whether every stage succeeded:

grep "ERROR" /var/log/app.log | wc -l
echo $?

wc -l almost always succeeds (exit 0), even if grep found zero matches or the log file didn’t exist — so this always prints 0, hiding a real failure. Two fixes: check the array PIPESTATUS, which holds the exit status of every command in the last pipeline, or turn on pipefail so the pipeline’s overall status becomes the first nonzero exit code in it:

set -o pipefail
grep "ERROR" /var/log/app.log | wc -l
echo "${PIPESTATUS[@]}"

Best Practices

  • Avoid piping a file through cat just to hand it to a command that can read files itself — pass the filename as an argument instead.
  • When a pipeline needs to capture error output as well as normal output, put 2>&1 right before the pipe symbol on the command whose errors you want captured.
  • Add set -o pipefail near the top of scripts that use pipelines, so a failure in an early stage isn’t silently masked by a later stage’s success.
  • Use tee when you need to both watch output live and save it to a file, rather than running the same command twice.
  • Break long pipelines across multiple lines (ending each line with | before the newline) so each stage is easy to read and debug independently.
  • Use && to require every prior step to succeed before a risky or destructive step runs, and || to provide a fallback or clear failure message.
  • When a chained command line gets longer than two or three links, consider rewriting it as a small script with an explicit if statement for readability.

Practice Exercises

Exercise 1: Write a single pipeline that lists the files directly inside /etc, keeps only the ones with “conf” in the name, and prints just the count of matches. Hint: combine ls, grep, and wc -l.

Exercise 2: Using /var/log/auth.log, build a pipeline that finds lines containing “Failed password” and reports the five source IP addresses that appear most often, similar to the nginx example above but adjusting which field you cut.

Exercise 3: Write a chained command line that runs ./deploy.sh, prints “Deploy OK” if it succeeds, and otherwise prints “Deploy FAILED” to stderr and causes the shell to report a nonzero exit status. Then verify with $? or PIPESTATUS that a deliberate failure is not silently hidden.

Summary

  • A pipe (|) connects one process’s stdout directly to the next process’s stdin via a kernel buffer, with no intermediate file.
  • Pipelines run all their commands concurrently, so downstream commands can start consuming data before upstream commands finish producing it.
  • A pipe never touches stderr by default — use 2>&1 before the pipe if you need error output captured too.
  • && and || chain commands based on exit status (0 = success), while ; just runs commands in sequence regardless of outcome.
  • After a pipeline, plain $? only reflects the last command; use PIPESTATUS or set -o pipefail to catch failures earlier in the chain.
  • Avoid the useless-use-of-cat pattern — let the first real command in a pipeline read the file directly when it can.