grep, sed, and awk: Reliable Text Search and Small Stream Programs

Select before transforming

grep filters lines matching a pattern. Extended regular expressions with grep -E make alternatives and repetition easier to read. Use -F for literal text; it avoids interpreting metacharacters when the search string is not intended as a regular expression.

grep -nE 'ERROR|WARN' application.log

Quote patterns to protect spaces, wildcard characters, and shell expansions. For logs with binary-looking content, grep -a changes interpretation; choose that intentionally rather than suppressing encoding problems without thought.

sed for stream transformations

sed applies editing commands as input streams past. A substitution such as sed 's/old/new/g' file writes transformed text to standard output and leaves the source unchanged. In-place options differ across GNU and BSD/macOS sed; verify the local implementation and make a backup before automating edits across files.

Avoid building a complex migration from one opaque regular expression. Start with a representative fixture, inspect a diff, then run on the full input. Escaping rules occur in both the shell and the sed expression, so single-quote static programs where possible.

awk for records and fields

awk splits each input record into fields and lets a small program select and format rows. The default field separator handles whitespace-delimited text; CSV with quoted commas is not safely parsed by default awk field splitting. Use a real CSV parser when quoting and escaping rules matter.

awk '$3 >= 500 { print $1, $3 }' access.log

This example selects records whose third whitespace-delimited field is at least 500. Real web logs may have quoted fields, embedded spaces, or different layouts; confirm the format before relying on numeric field positions.

Compose pipelines defensively

set -o pipefail
producer | grep -F 'target' | awk '{ print $1 }'

Without pipefail, many shells report only the final command's status. Also distinguish “no matches” from a real grep error: grep commonly exits 1 when no line matches and 2 for an error. Scripts should handle expected absence separately from broken input or permissions.

When a pipeline becomes a program

Use a small script or a tested parser when data has nested structure, quoting rules, multiline records, or security implications. Text filters are excellent for diagnostics and simple transformations; they are not substitutes for structured parsers. Make assumptions about delimiters, locale, line endings, and input size explicit.