Delivery Engineering Advanced: Build Caches, Runner Isolation, and Safe Releases

1. Docker layer caching without surprises

A Docker build cache is invalidated from the first changed instruction onward. If COPY . . happens before dependency installation, changing one source line invalidates the dependency layer too. Copy lockfiles and dependency manifests first, install reproducibly, then copy application source:

FROM golang:1.24 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/service ./cmd/service

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/service /service
ENTRYPOINT ["/service"]

Use docker build --cache-from ... only with compatible cache metadata and trusted sources; a cache should save time, not decide correctness. docker system prune --volumes can delete volumes containing useful or irreplaceable data. Inspect targets and backups before pruning; on shared CI hosts, use isolated ephemeral workers instead of sweeping shared storage.

docker build --cache-from registry.example.com/team/service:main \
  --tag registry.example.com/team/service:"$GIT_SHA" .

# Inspect reclaimable space first. Review every target before pruning volumes.
docker system df -v
# Destructive: may delete unused volumes and their data.
docker system prune --volumes

For compiled services, keep toolchains out of the runtime stage. The Go example above copies one static binary; Rust follows the same pattern by copying target/release/service, while Java commonly builds a JAR with Maven/Gradle and copies only that JAR into a JRE runtime image. Pin both stage images and test the final runtime stage, not just the builder.

2. Runner and container isolation

A self-hosted runner executes code from repositories and pull requests. Treat it as a workload execution boundary: use ephemeral runners, isolate credentials by job, restrict network access, and do not share writable caches across trust levels. Container user namespaces map container identities to host identities; Linux capabilities grant subsets of privileged operations.

--privileged broadly relaxes isolation and grants extensive capabilities/devices. CAP_SYS_ADMIN itself is very broad. Prefer dropping all capabilities and adding only specific ones needed, with seccomp/AppArmor/SELinux and read-only filesystems where practical. A shared multi-tenant build worker should never assume a container is a complete security boundary.

Generate an SBOM, scan images with a tool such as Trivy, sign trusted artifacts with Cosign, and verify signatures at deploy time. Pin base images by digest for reproducibility and define how findings are triaged; a scanner without ownership or policy is only a report generator.

3. Diagnose exit code 137

Exit 137 conventionally means the process received SIGKILL (128 + signal 9), but it does not prove an OOM kill. Inspect container runtime state, kernel/cgroup OOM events, memory.current/memory.events on cgroup v2, Kubernetes pod termination reason, node pressure, and application logs. A supervisor or operator may also send SIGKILL after a grace period. Fix memory growth or limits only after identifying who sent the signal and why.

4. Expand-contract database migrations

For rolling deployments, first add a backward-compatible nullable column or new table (expand), deploy code that can read old and new forms, backfill safely, switch writes/reads, then remove the old schema only after all old binaries are gone (contract). Dropping a column in the same release that first stops using it is unsafe when old pods remain active.

Backfills should be resumable, rate-limited, observable, and chunked. Define rollback behavior at every stage; a migration may be forward-only once new data has been written.

5. Terraform/OpenTofu state locks

A state lock prevents concurrent writers from corrupting the state transition. If a pipeline crashes, first verify no apply is still active and inspect the backend lock metadata and owning run. force-unlock is a last resort, only after confirming the holder is stale and coordinating with the team; unlocking a live apply permits concurrent state mutation. Preserve the state backup and reconcile cloud reality with the state before continuing.

Interview and troubleshooting drills

Interview: Why can one source edit invalidate all following Docker layers? Walk through instruction cache keys and show how copying manifests before source keeps dependency installation reusable.

Troubleshooting: A container exits 137. Distinguish kernel/cgroup OOM from operator SIGKILL by checking termination reason, cgroup events, node pressure, and shutdown timing before changing memory limits.

Challenge: In a multi-tenant runner, compare --privileged with dropping all capabilities and adding one narrow capability. Explain why user namespaces reduce host impact but do not replace runner isolation, least-privilege credentials, and network restrictions.