Text Processing

The tools in this chapter turn text into structured data. They are the building blocks of the Unix philosophy: each one does one thing well, and they are connected with pipes (|).

grep — search for text

grep prints lines that match a pattern.

$ grep apple fruits.txt
apple
apple
apple

Common flags:

$ grep -n apple fruits.txt
1:apple
4:apple
6:apple

grep -r searches a whole directory tree:

$ grep -r "error" /var/log/

Combine -r and -l to list which files contain a match, without printing the lines themselves:

$ grep -rl "error" /var/log/

sed — stream editor

sed edits text line by line. The most common use is find-and-replace.

$ sed "s/apple/pear/" fruits.txt
pear
banana
cherry

sed only changes the first match per line; add g to replace all:

$ sed "s/apple/pear/g" fruits.txt

Print a specific line with -n and a line address:

$ sed -n "2p" fruits.txt
banana

Delete a line with d:

$ sed "2d" fruits.txt

To edit a file in place (no output to screen), use -i:

$ sed -i "s/cherry/peach/" fruits.txt

Careful: sed -i overwrites the original file — make a backup with -i.bak if you are unsure.

awk — field-based processing

awk works on lines split into fields. By default fields are separated by whitespace and numbered from $1.

$ printf "alice 10\nbob 20\n" | awk '{print $1}'
alice
bob

(printf prints with escapes — \n is a newline. It is the predictable cousin of the echo -e from Chapter 4, and the traditional choice inside pipelines.)

Print a column by number:

$ awk '{print $2}' scores.txt

With a custom separator, use -F:

$ echo "a:b:c" | awk -F: '{print $2}'
b

awk can compute over lines with END:

$ awk '{sum += $2} END {print sum}' scores.txt
30

cut — extract columns

cut pulls out fields by delimiter. Specify the delimiter with -d and the fields with -f:

$ printf "a,b,c\nd,e,f\n" | cut -d, -f2
b
e

sort — order lines

sort orders lines alphabetically.

$ sort fruits.txt
apple
apple
apple
banana

$ sort -n numbers.txt
$ sort -r fruits.txt

uniq — remove duplicates

uniq removes adjacent duplicate lines. Since duplicates are usually not adjacent, it is almost always combined with sort:

$ sort fruits.txt | uniq
apple
banana
cherry
date

Count occurrences with -c:

$ sort fruits.txt | uniq -c
      3 apple
      1 banana
      1 cherry
      1 date

wc — word count

wc counts lines, words and bytes.

$ wc fruits.txt
 6  6 37 fruits.txt

Use -l for lines only:

$ wc -l fruits.txt
6 fruits.txt

A common pattern: count how many files are in a directory:

$ ls | wc -l

More text tools

A few more tools that appear everywhere in shell workflows:

diff shows the differences between two files:

$ diff old.txt new.txt

tac prints a file in reverse line order (it is cat backwards):

$ tac notes.txt

nl numbers the lines of a file:

$ nl notes.txt
     1  first line
     2  second line

tr translates characters — swap one set for another, or squeeze repeats down to one:

$ echo hello | tr a-z A-Z
HELLO
$ echo "a  b  c" | tr -s " "
a b c

column -t aligns columns of text into a neat table:

$ printf "alice 10\nbob 20\n" | column -t
alice  10
bob    20

watch re-runs a command repeatedly and shows the changing output on screen — handy to watch a value over time:

$ watch free -h

Press Ctrl+C to stop watch.

comm compares two sorted files, showing lines unique to each and lines in both:

$ comm sorted1.txt sorted2.txt

xargs turns its standard input into arguments for another command. It is the power tool for acting on a list of items, such as files found by find:

$ find . -name "*.log" | xargs rm

Putting it together

These tools compose naturally. For example, the five most common words in a file:

$ tr -s ' ' '\n' < guide.txt | sort | uniq -c | sort -nr | head -5

Reading it left to right: tr turns every space into a newline (one word per line), and the < feeds guide.txt into it — input redirection, chapter 13’s > with the arrow flipped. Then sort puts neighbouring words together, uniq -c counts them, and sort -nr orders by count.