awk: Pattern Scanning Basics

awk is a text-processing language and command-line tool built around a simple idea: scan a file line by line, test each line against a pattern, and run an action when the pattern matches. It ships on every Linux system and is one of the most powerful tools for working with structured, column-based text — log files, CSV-like data, /etc/passwd, output from other commands — without writing a full program. Where grep only finds matching lines and sed only edits text, awk understands a line’s structure as a set of fields, so you can select, transform, and compute over columns of data directly from the shell.

Overview: How awk Works

awk (named after its creators Aho, Weinberger, and Kernighan) is built around records and fields. By default, awk treats each line of input as one record, and splits that record into fields wherever it finds one or more spaces or tabs. The whole line is available as $0; the first field is $1, the second is $2, and so on, up to the last field, $NF, where NF (Number of Fields) is a built-in variable holding the field count for the current record. Another built-in variable, NR (Number of Records), counts how many lines awk has read so far — it behaves like a running line number.

A complete awk program is a sequence of pattern { action } pairs. For every record awk reads, it checks each pattern in order; if a pattern matches (or is left out, which matches every line), the associated action runs. A pattern can be a regular expression, a numeric or string comparison, or a special keyword: BEGIN runs its action once before any input is read (useful for setting up variables or printing headers), and END runs its action once after all input has been processed (useful for printing totals). If you write only a pattern with no action, awk performs the default action, which is {print $0} — print the whole matching line, much like grep.

Internally, awk works as a loop: read one record, split it into fields using the field separator FS (default: whitespace), evaluate every pattern-action pair against that record, then move to the next record. Because the same variables (like a running sum, or a counter you increment) stay alive from one line to the next, tasks like totalling a column or counting matches feel natural in awk — something grep and sed cannot do on their own.

Syntax

The general form of an awk invocation is shown below (this is a syntax diagram, not a command to run as-is — replace pattern, action, and file with real values):

awk 'pattern { action }' file

A fuller program can include BEGIN and END blocks, and multiple pattern-action pairs:

awk 'BEGIN { setup }
     pattern1 { action1 }
     pattern2 { action2 }
     END { cleanup }' file

Always wrap the awk program in single quotes so the shell passes $1, $2, and similar references straight to awk instead of trying to expand them itself. Common command-line options and built-in variables:

Item Meaning
-F fs Set the field separator (e.g. -F: for colon-delimited files like /etc/passwd)
-v var=value Set an awk variable before the program runs
-f file.awk Read the awk program from a file instead of the command line
$0 The entire current record (line)
$1 … $NF Individual fields of the current record
NF Number of fields in the current record
NR Number of the current record (running line count)
FS Input field separator (default: whitespace)
OFS Output field separator used by print when joining fields with commas
FILENAME Name of the file currently being read

Examples

Example 1: Print specific fields from a colon-delimited file

/etc/passwd stores one user account per line, with fields separated by colons. Field 1 is the username and field 7 is the user’s login shell:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
maria:x:1000:1000:Maria Lopez:/home/maria:/bin/bash
awk -F: '{print $1, $7}' /etc/passwd

Output:

root /bin/bash
daemon /usr/sbin/nologin
sync /bin/sync
maria /bin/bash

The -F: option tells awk to split each line on colons instead of whitespace. With that field separator, $1 is the username and $7 is the shell field; the comma in print $1, $7 joins them with a single space, the default OFS.

Example 2: Filter rows by a numeric field

Suppose a simplified web server log at /var/log/app/access.log has space-separated fields: date, time, client IP, HTTP method, path, status code, and bytes sent:

2026-08-04 10:15:32 203.0.113.5 GET /index.html 200 512
2026-08-04 10:15:35 203.0.113.9 POST /login 500 128
2026-08-04 10:16:01 203.0.113.5 GET /about.html 200 1024
2026-08-04 10:16:20 198.51.100.7 GET /missing.html 404 0
awk '$6 >= 400 {print $3, $6}' /var/log/app/access.log

Output:

203.0.113.9 500
198.51.100.7 404

Field 6 ($6) holds the status code. The pattern $6 >= 400 is checked against every record; only the two lines with server or client errors match, so the action {print $3, $6} only runs for those, printing the client IP and status code.

Example 3: Total a column with BEGIN/END

awk '{sum += $7} END {printf "Total bytes: %d\n", sum}' /var/log/app/access.log

Output:

Total bytes: 1664

Here the action {sum += $7} has no pattern, so it runs on every line, adding field 7 (bytes sent) to the variable sum. Because awk variables persist across records, sum keeps accumulating as each line is read. The END block runs once, after the last record, and printf formats the total with an explicit newline (\n), unlike print, which adds a newline automatically but gives you no control over formatting.

How awk Processes Input, Step by Step

  1. Before reading any input, awk runs every BEGIN block, in the order they appear.
  2. awk reads one line of input and stores it in $0; it increments NR by one.
  3. awk splits $0 into fields using the current field separator FS, filling in $1 through $NF and setting NF to the field count.
  4. awk evaluates each pattern in the program, top to bottom, against the current record. For every pattern that matches, it runs the associated action immediately, then checks the next pattern — a single line can trigger several actions if it matches several patterns.
  5. Once all patterns have been checked, awk goes back and reads the next line. This repeats until the input is exhausted.
  6. After the last line has been processed, awk runs every END block, in the order they appear, then exits.

This read-split-match-act loop is why awk is described as a pattern scanning and processing language: you almost never write an explicit loop yourself — awk supplies the loop over lines, and you just describe what should happen on each one.

Common Mistakes

Mistake 1: Double-quoting the awk program

Wrapping the program in double quotes lets the shell — not awk — expand $1 before awk ever sees it. Since a plain shell usually has no $1 set, the field reference silently disappears:

awk "{print $1}" file.txt

Use single quotes so the dollar signs reach awk untouched:

awk '{print $1}' file.txt

Mistake 2: Using = instead of ==

A single = is assignment, not comparison. Used in a pattern, it assigns a value to the field on every line — and the assignment itself is truthy — so every line matches, and the field’s original value is overwritten:

awk '$6 = 500 {print}' /var/log/app/access.log

Use == to compare instead of assign:

awk '$6 == 500 {print}' /var/log/app/access.log

Mistake 3: Forgetting to set the field separator

awk’s default field separator is whitespace. Running an awk program against a colon-delimited file like /etc/passwd without -F: means there is no whitespace to split on, so the entire line becomes $1:

awk '{print $1}' /etc/passwd

Set the field separator explicitly to match the file’s actual delimiter:

awk -F: '{print $1}' /etc/passwd

Best Practices

  • Always wrap the awk program in single quotes so field references like $1 and $NF reach awk instead of being expanded by the shell.
  • Set -F explicitly whenever the input isn’t whitespace-separated — don’t rely on the default.
  • Use printf instead of print when you need specific formatting, such as fixed decimal places or no trailing newline.
  • Use a pattern like NR==1 {next} to skip a header line before processing data rows.
  • Prefer $NF and $(NF-1) over hardcoded field numbers when the field count can vary between lines.
  • For programs longer than a line or two, save them in a .awk file and run them with awk -f script.awk file — far easier to read and version-control than a long one-liner.
  • Reach for awk when you need to select, compare, or compute across columns; reach for grep when you only need to find matching lines, and sed when you only need to substitute text.

Practice Exercises

  1. Using /etc/passwd, print the usernames of every account whose user ID (field 3) is 1000 or greater — these are typically the regular human user accounts, as opposed to system accounts. Hint: you’ll need -F: and a numeric comparison pattern.
  2. Using a log file like the one in Example 2, count how many requests returned status 200. Hint: use a pattern to match the status field, increment a counter variable in the action, and print the counter in an END block.
  3. Write an awk command that prints the last field of every line in a file, without knowing in advance how many fields each line has. Hint: NF always holds the current line’s field count.

Summary

  • awk scans input one record (line) at a time, splits each record into fields, and runs an action for every pattern that matches.
  • $0 is the whole line, $1 through $NF are individual fields, NF is the field count, and NR is the running record number.
  • -F sets the field separator; the default is whitespace, so remember to change it for delimited files like /etc/passwd.
  • BEGIN runs once before input is read; END runs once after all input is processed — useful for setup and totals.
  • Always single-quote the awk program so the shell doesn’t expand $1-style field references itself.
  • Use == for comparison; a bare = assigns and will make every pattern match.