Kubernetes disruption budgets: protect capacity, not every failure

First principles

A maintenance operation and a hardware failure can both remove a pod, but they enter Kubernetes through different paths. During a normal node drain, the client requests eviction through an admission-aware API. A PodDisruptionBudget, or PDB, can refuse that request when too little healthy capacity would remain. A sudden node failure cannot ask permission before it occurs. The budget therefore constrains supported voluntary disruptions; it does not reserve functioning hardware or guarantee continuous availability.

Start with the service's capacity requirement. If one healthy replica can handle only half the expected traffic, retaining one out of three pods is insufficient even though the service still has an endpoint. A budget should express a defensible minimum healthy population, with headroom for real traffic and dependency limits. Health accounting normally uses pod readiness, which makes the quality of your readiness signal central to the usefulness of the budget. A ready pod that cannot serve requests inflates apparent safety.

With three desired replicas and minAvailable: 2, three healthy pods usually permit one eviction. Once one pod becomes unavailable, further eviction must wait for recovery. This is not a sequence lock attached permanently to a particular node. Controllers continually recompute status from selected pods and workload information. Inspect status before maintenance rather than calculating permission once and assuming it remains valid while deployment, scaling, and unrelated failures happen concurrently.

Runnable example

Use a disposable cluster where the nginx:1.27-alpine image is available, and save this as budget-demo.yaml. The manifest uses a dedicated namespace and a narrow selector. It deliberately does not enforce node spreading, which leaves placement as a separate condition to inspect.

apiVersion: v1
kind: Namespace
metadata:
  name: budget-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: budget-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: budget-web
  template:
    metadata:
      labels:
        app: budget-web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 2
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: budget-demo
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: budget-web

Apply with kubectl apply -f budget-demo.yaml, wait using kubectl rollout status deployment/web -n budget-demo, and inspect kubectl get pdb web -n budget-demo -o yaml. Once all replicas are ready, expect currentHealthy to be three, desiredHealthy to be two, and disruptionsAllowed to be one. These are observed status values, not instant guarantees; controller reconciliation can briefly lag the newest pod state.

Contextual failure case

A platform team starts a node upgrade while a service already has one unready replica. Its budget requires two healthy replicas out of three, so the drain refuses an additional eviction. An operator removes the PDB to finish maintenance. Now the node disappears with another healthy replica, leaving only one to absorb all traffic. The budget was identifying an existing recovery problem, not causing the original outage. Investigate the unready pod, placement, available node resources, and dependency health first.

There is also a misleading opposite case. A PDB allows an eviction, yet the replacement cannot schedule because every remaining node lacks memory. The budget measured current health; it did not promise spare scheduling capacity. A safe maintenance plan needs both an availability constraint and a recovery path. Topology spread, anti-affinity, resource requests, and autoscaler behavior belong to that recovery analysis. More restrictive placement can improve failure isolation while making replacement harder on a small cluster.

Verification

In the disposable namespace, scale the deployment to two replicas and wait for both to become ready. The same budget should report zero allowed disruptions. Scaling is useful here precisely because workload scale-down is not prevented by the PDB in the way an eviction is. Restore three replicas and observe the allowance return. Do not drain a shared or production node to test this lesson. Keep an incident timeline of readiness changes, eviction responses, and scheduling events if maintenance stalls.

Exercise

Write a maintenance preflight checklist for a service with four replicas that requires three healthy replicas at peak traffic. Include observed healthy count, failure-domain placement, replacement capacity, and how to stop maintenance when another replica fails. Then explain why raising desired replicas without checking dependency connection limits can trade one outage mechanism for another. Your answer should distinguish the arithmetic of permitted evictions from the operational conditions that make recovery possible.

Safety and caveats

Direct deletion, involuntary failures, and workload-controller rollout behavior are not universally blocked by a PDB. Configure deployment rollout availability separately. Percentage budgets involve rounding, so test small replica counts rather than assuming intuitive fractions. Kubernetes versions also differ in supported unhealthy-pod eviction policies; inspect your cluster before relying on one. Remove only this lesson's dedicated namespace when finished. Never weaken a production budget solely to clear a maintenance error without documenting the availability risk and recovery plan.