Unix Log Sorting: Normalize Offset Timestamps Before Ordering Events

A text sort reverses the requests

One service writes 2026-10-06T09:30:00+02:00; another writes 2026-10-06T08:00:00Z. The first event happened at 07:30Z, before the second. Sorting the original strings puts 08:00 before 09:30 and tells the wrong story. ISO-looking text is not automatically a universal chronological sort key.

Even all-UTC timestamps need a consistent representation. Mixed fractional precision, missing date fields, localized month names, and unpadded values complicate lexical order. A log line containing a user's quoted timestamp introduces another trap: searching for the first date-like token may select payload text rather than the emitter's timestamp.

Inspect the producer format first

For an incident bundle, define a line format before building a pipeline. This example expects newline-delimited JSON with a top-level event_at string and an optional message. It targets Python 3.11+ for offset-aware ISO parsing and runs the parser as a normal script, rather than depending on GNU date flags that are not portable to BSD/macOS date.

Save the example as an investigation script in your own working area, then run it against a copied evidence file. The article does not prescribe editing a live log file in place.

import json
import sys
from datetime import datetime, timezone
from decimal import Decimal

epoch = datetime(1970, 1, 1, tzinfo=timezone.utc)

for line_number, raw in enumerate(sys.stdin, start=1):
    try:
        record = json.loads(raw)
        parsed = datetime.fromisoformat(record["event_at"])
        if parsed.tzinfo is None or parsed.utcoffset() is None:
            raise ValueError("missing offset")
        delta = parsed.astimezone(timezone.utc) - epoch
        micros = ((delta.days * 86400 + delta.seconds) * 1_000_000
                  + delta.microseconds)
        seconds = Decimal(micros) / Decimal(1_000_000)
        encoded = json.dumps(record, ensure_ascii=True, separators=(",", ":"))
        print(f"{seconds}\t{line_number}\t{encoded}")
    except (ValueError, TypeError, KeyError, OverflowError) as error:
        print(f"line {line_number}: {error}", file=sys.stderr)

This uses integer timedelta components before decimal formatting, avoiding float rounding during timestamp conversion. The input line number is a deterministic tie-breaker, not evidence that simultaneous events have a causal order. JSON serialization escapes embedded tabs and newlines inside strings so the derived tab-separated record stays one physical line.

Sort the derived evidence

python3 normalize_logs.py < incident.ndjson > normalized.tsv 2> rejected.log
LC_ALL=C sort -t "$(printf '\t')" -k1,1n -k2,2n normalized.tsv > ordered.tsv
cut -f3- ordered.tsv > ordered.ndjson
wc -l incident.ndjson normalized.tsv rejected.log

Numeric sorting matters for negative epoch values and decimal seconds. Setting LC_ALL=C removes locale-dependent surprises. Keep the original bundle and the rejected-record report together. A successful sort command does not prove every input record was parsed, and the illustrative parser intentionally continues after individual errors. Decide whether any rejection should invalidate your incident view before relying on it.

Bound the operational cost

Text sorting may spill to temporary disk when the input exceeds memory. Work on a bounded incident window, check available space, and use an appropriate temporary directory with restricted permissions if the data is sensitive. Avoid dumping a multi-gigabyte production bundle into a workstation's small system partition. Keep access-controlled evidence separate from shareable redacted extracts.

Do not pass untrusted log text through eval or construct shell commands from parsed fields. Pipes carry data; log payloads are not shell instructions. A message containing a dollar sign or backtick must remain a string throughout the investigation.

Interpret the sorted result cautiously

Sorting normalizes representation, not clock accuracy. A skewed producer can still appear earlier than the request that triggered it. Compare trace IDs, receipt timestamps, host-clock evidence, and local sequence numbers before making causal claims. For a small extract, the timeline utility provides another inspection view; preserve line errors and ties rather than editing them away to make the display neat. Test the pipeline with mixed offsets, equal instants, invalid JSON, a missing offset, and a record whose message contains its own unrelated timestamp.