Sorting and Counting (sort, uniq, wc)

sort, uniq, and wc are three small coreutils that turn raw text into organized, countable information. On their own they each do one narrow job; piped together they answer questions like “what are my ten busiest IP addresses today?” in a single line. This lesson covers how each tool actually works, how to combine them, and the mistakes that trip up almost everyone the first time.

Overview / How it works

All three tools read lines of text — from a file argument or from standard input — and write results to standard output. That shared interface is what lets you chain them with the pipe operator |, which asks the kernel to connect one process’s stdout file descriptor directly to the next process’s stdin file descriptor. Both processes run concurrently; the kernel buffers data between them so the reader doesn’t need the writer to finish first.

sort reorders the lines of its input. By default it compares lines byte-by-byte according to the current locale’s collation rules (roughly alphabetical, with case and locale affecting exact ordering), not numerically and not by “natural” order. For huge files that don’t fit in memory, GNU sort doesn’t choke — it performs an external merge sort: it sorts manageable chunks in memory, writes each sorted chunk to a temporary file (normally under /tmp), then merges those sorted chunks together. You never need to think about this in normal use, but it explains why sort can handle multi-gigabyte log files without running out of RAM.

uniq is deliberately simple: it is a streaming filter that compares each line only to the line immediately before it. It has no memory of lines seen further back, so it can only collapse or count adjacent duplicate lines. That single fact explains almost every uniq-related bug: if your data isn’t already sorted, identical lines that aren’t next to each other will not be treated as duplicates.

wc (“word count”) streams through its input counting three things as it goes: newline characters (-l, so it’s really counting line terminators, not “lines” in some abstract sense), whitespace-delimited words (-w), and bytes (-c) or characters (-m, which can differ from bytes in multibyte encodings like UTF-8). Because it’s a single pass over the stream, wc -l on a 10 GB log file is fast and uses almost no memory.

Syntax

sort [OPTIONS] [FILE...]
uniq [OPTIONS] [INPUT [OUTPUT]]
wc [OPTIONS] [FILE...]
Command Option Meaning
sort -n Compare fields as numbers, not text
sort -r Reverse the sort order
sort -u Output only the first line of each set of equal lines (sort + dedupe)
sort -f Fold case — treat uppercase and lowercase as equal
sort -k N Sort by field N instead of the whole line
sort -t C Use character C as the field delimiter (default: whitespace)
sort -h Compare human-readable sizes like 2K, 1G
uniq -c Prefix each output line with its repeat count
uniq -d Only print lines that appeared more than once
uniq -u Only print lines that appeared exactly once
uniq -i Ignore case when comparing lines
wc -l Count newline characters (lines)
wc -w Count words
wc -c Count bytes
wc -m Count characters (locale-aware)
wc -L Print the length of the longest line

Examples

1. Alphabetical and numeric field sorting

deploy_hosts.txt lists hostnames in the order they were added, not alphabetically:

sort deploy_hosts.txt

Output:

db01.internal
db02.internal
web01.internal
web02.internal
web03.internal

Reversing the order is just -r:

sort -r deploy_hosts.txt

Output:

web03.internal
web02.internal
web01.internal
db02.internal
db01.internal

Now suppose sales_by_region.csv holds comma-separated region totals: West,120000, East,95000, North,150000, South,80000. To rank regions by revenue, tell sort the delimiter is a comma, sort on the second field, treat it as a number, and reverse for highest-first:

sort -t',' -k2 -n -r sales_by_region.csv

Output:

North,150000
West,120000
East,95000
South,80000

Without -n, sort would compare the numbers as text and put 150000 before 80000 because '1' sorts before '8' as a character.

2. Removing and counting duplicates with uniq

uniq needs sorted input to work correctly, so it’s almost always paired with sort. To find the busiest client IPs in an Nginx access log, extract the first field of every line, sort those IPs, count adjacent runs, then sort the counts:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -5

Output:

   842 203.0.113.5
   401 198.51.100.23
   377 203.0.113.19
   205 198.51.100.4
    98 203.0.113.42

uniq -c also has narrower siblings. Given a sorted file visitor_ids_sorted.txt of user IDs, -d shows only IDs that visited more than once, and -u shows only IDs that visited exactly once:

uniq -d visitor_ids_sorted.txt
uniq -u visitor_ids_sorted.txt

Output:

1024
3391

1002
1017
4420

3. Counting lines, words, and bytes with wc

A quick way to see how large a log file has grown:

wc -l /var/log/syslog

Output:

128453 /var/log/syslog

Pass several files and wc prints a per-file breakdown plus a total:

wc access.log app_errors.log

Output:

   9832   68824  1184032 access.log
    214    1930    24011 app_errors.log
  10046   70754  1208043 total

Combining sort -u with wc -l is a common way to count distinct values, such as how many unique customer emails appear in an export:

sort -u customer_emails.txt | wc -l

Output:

482

How it works step by step

Walking through awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -5:

  • The shell starts five processes and wires each stage’s stdout to the next stage’s stdin using pipes — kernel-managed buffers identified by file descriptors, not temporary files.
  • awk reads the log line by line and prints just the first whitespace-delimited field (the client IP), streaming results out as soon as each line is processed.
  • sort reads that entire stream of IPs, buffering (and spilling to temp files if needed) until it has seen everything, because it can’t know the final order until the last line arrives; it then writes IPs out in ascending order, so identical IPs are now adjacent.
  • uniq -c streams through the sorted IPs one line at a time, counting consecutive repeats and emitting one count ip line per distinct run.
  • The second sort -nr reorders those count-prefixed lines numerically, largest count first, ignoring the leading whitespace that uniq -c pads onto small numbers.
  • head -5 reads only the first five lines it receives and then closes its input early, which causes the earlier stages to receive a broken-pipe signal and exit once their buffered output is flushed.

Common Mistakes

Mistake 1: Running uniq on unsorted input.

uniq contacts.txt

If contacts.txt has “alice@example.com” appearing on line 2 and again on line 40, this does nothing to remove the second one — uniq only compares each line to its immediate predecessor. Sort first:

sort contacts.txt | uniq

Mistake 2: Forgetting -n for numeric data.

sort file_sizes.txt

If the file contains 4, 33, 2, and 10, plain sort compares them as text and produces 10, 2, 33, 4 — every line starting with '1' sorts before one starting with '2'. Use numeric comparison:

sort -n file_sizes.txt

Mistake 3: Assuming sort -u ignores case.

sort -u names.txt

If names.txt contains both Alice and alice, sort -u keeps both by default, since uppercase and lowercase letters have different byte values and are not considered equal. Fold case explicitly when that’s the intent:

sort -u -f names.txt

Best Practices

  • Use sort -u instead of sort file | uniq when you only need a sorted, deduplicated list — it’s one process instead of two.
  • Always sort before piping into uniq unless you know for certain the input is already sorted (for example, output from another sort).
  • Add -n whenever you’re sorting anything that represents a quantity — sizes, counts, timestamps stored as numbers — never rely on the default text comparison for numbers.
  • When sorting delimited data, be explicit with both -t (the delimiter) and -k (the field), rather than assuming the default whitespace split matches your data.
  • Set LC_ALL=C before sort when you need a fast, byte-value-only ordering that doesn’t depend on the system locale — useful in scripts that must behave identically on every machine.
  • Remember wc -l counts newline characters: a file whose last line has no trailing newline will be undercounted by one.
  • Quote filenames and variables in every command (sort "$input_file") so names containing spaces don’t get split into multiple arguments.

Practice Exercises

  • You have a file order_totals.txt with one dollar amount per line, unsorted. Write a command that prints the five largest order totals. Hint: you need -n, -r, and head.
  • You have signup_emails.txt where the same email may appear multiple times across the file in no particular order. Produce a count of exactly how many times each unique email appears, sorted from most to least frequent. Hint: this is the sort | uniq -c | sort -nr pattern.
  • You have error_codes.log, one HTTP status code per line. Find out how many distinct status codes appear in the file, as a single number. Hint: combine sort -u with wc -l.

Summary

  • sort reorders lines; add -n for numbers, -r to reverse, -k/-t to sort on a specific delimited field.
  • uniq only collapses or counts adjacent duplicate lines, so input must be sorted first — that’s why sort | uniq is almost always seen together.
  • uniq -c prefixes each line with its repeat count; combine with a second sort -nr to rank the most frequent values.
  • wc -l, -w, and -c count newlines, words, and bytes respectively, streaming through input in a single fast pass.
  • sort -u file | wc -l is a quick idiom for counting distinct values in a file.
  • Text comparison is byte/locale based by default — always use -n for numeric data and -f/-i when case shouldn’t matter.