Release Safety Advanced: State, Schema Compatibility, and Artifact Trust

1. Immutable infrastructure and safe change

Immutable infrastructure replaces instances or artifacts rather than editing unknown state in place. This improves reproducibility only when configuration, artifact identity, and rollout strategy are controlled. Record the source revision, build provenance, runtime configuration, and migration version for each release.

Schema changes need compatibility across overlapping application versions. Expand-contract migrations add a backward-compatible field, deploy code that can work with both forms, backfill and switch use, then remove the old field in a later release. A rolling deployment is not atomic: old and new pods may issue queries against the same database concurrently.

2. CI runners and supply chain

Self-hosted runners execute repository-controlled code. Use ephemeral isolated workers, short-lived credentials, per-job permissions, and trust-separated caches. An SBOM records components; Trivy or a comparable scanner identifies known vulnerabilities; Cosign can sign and verify artifacts. Pin base images and actions where practical, and define who owns exceptions and remediation. A scan result without policy and provenance does not establish trust.

Docker layer cache keys depend on prior instruction inputs. Copy dependency manifests before volatile source to preserve reusable layers, but never treat --cache-from as proof that an artifact is correct. A clean build should remain valid.

3. Terraform/OpenTofu state-lock recovery

State locks prevent two writers from applying changes based on competing snapshots. When a lock remains after a crashed job, identify the lock owner and verify that no apply process is still running. Preserve a state backup and compare the backend state with real infrastructure. Only then consider force-unlocking the specific stale lock; removing a live lock can allow concurrent state writes and corrupt tracking.

After recovery, run a plan and review drift before applying. Do not use force-unlock as a generic retry button.

4. Container privilege and capability boundaries

--privileged grants broad host-facing capabilities and devices. Prefer a non-root user, drop capabilities by default, then add only those required. CAP_NET_ADMIN permits network administration functions and is still powerful; it should be granted only to workloads that need it. User namespaces map container identities to host IDs and reduce some impact, but do not replace host patching, seccomp, LSM policy, and isolation between untrusted build jobs.

Interview and troubleshooting drills

Interview: Explain expand-contract for dropping a column during a rolling deployment. Name the compatibility period, backfill verification, rollout completion check, and delayed contract migration.

Troubleshooting: Terraform apply left a stale lock after a runner crash. Verify no live apply, inspect lock metadata and backend version, preserve state, force-unlock only the confirmed stale holder, then plan and reconcile before another apply.

Challenge: Compare a privileged build container with a minimal capability set, user namespace, and ephemeral runner. Identify which controls limit kernel attack surface, credential exposure, and persistence between jobs; no single container flag supplies the entire trust boundary.