Text Processing: grep, sed, awk and Pipes

Linux tools do one thing each and pass text through pipes. Together they answer questions about log files and configuration in seconds, with no scripts to write. This post builds from simple searches to useful one-line reports, using a web server access log as the example.

Versions in use on this date. GNU grep 3.1, sed 4.4 and gawk 4.1.4 on Ubuntu 18.04.2 LTS. CentOS 7.6 has older versions (grep 2.20, sed 4.2.2, gawk 4.0.2), so avoid very new options when a script must run on both.

Pipeline model

flowchart LR
  F[access.log] --> G[grep<br/>select lines]
  G --> A[awk<br/>pick fields]
  A --> S[sort]
  S --> U[uniq -c<br/>count]
  U --> SR[sort -rn<br/>rank]
  SR --> H[head<br/>top ten]

Each stage reads standard input and writes standard output. Redirect with > (overwrite) and >> (append); 2>&1 joins errors to the output.

grep: find lines

grep -n "ERROR" app.log                # line numbers
grep -i "timeout" app.log              # ignore case
grep -v "healthcheck" access.log       # exclude
grep -E "HTTP/1.1\" (4|5)[0-9]{2}" access.log      # extended regex: 4xx and 5xx
grep -rn "listen" /etc/nginx/ --include='*.conf'
grep -c "POST" access.log              # count matching lines
zgrep "ERROR" app.log.2.gz             # search rotated, compressed logs
grep -B2 -A5 "Traceback" app.log       # context before and after

awk: work with columns

Default field separator is whitespace; $1 is the first field, $NF the last.

awk '{print $1}' access.log | head                       # client IPs
awk '$9 >= 500 {print $4, $7, $9}' access.log             # time, URL, status for server errors
awk '{sum += $10} END {print sum/1024/1024 " MB"}' access.log     # bytes served
awk -F: '{print $1, $7}' /etc/passwd                       # custom separator
awk '{c[$9]++} END {for (s in c) print s, c[s]}' access.log | sort

The last example counts requests per HTTP status without sort | uniq.

sed: edit streams and files

sed 's/foo/bar/g' file                     # replace all on every line
sed -n '10,20p' file                       # print lines 10 to 20
sed '/^#/d;/^$/d' /etc/ssh/sshd_config     # drop comments and blanks
sed -i.bak 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config

-i.bak edits in place and keeps a backup copy. Test the substitution without -i first, and validate the service config before reloading (sshd -t).

Useful companions

sort -k2 -n file            # sort by field 2 numerically
sort | uniq -c | sort -rn   # frequency table
cut -d, -f1,3 data.csv      # columns of a CSV (no quoted commas)
tr -s ' ' < file            # squeeze repeated spaces
tee out.txt                 # write to file and still pass along
xargs -n1 -P4 command       # run in parallel from a list
column -t                   # align output in a table

Real questions, one pipeline each

# Top 10 client IPs by request count
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# Most requested URLs returning 404
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head

# Failed SSH logins by source address (Ubuntu)
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

# Requests per minute
awk '{print substr($4, 2, 17)}' access.log | uniq -c | tail

Good habits

• Build a pipeline one stage at a time and look at the output at each step.
• Quote patterns so the shell does not expand * or $.
• Run LC_ALL=C sort on big files; it is much faster and gives predictable ordering.
• Keep a file of your favourite one-liners; this is the start of your troubleshooting toolkit.
• When a job grows past a screen of text, move it into a script (see the shell scripting post).