Kubernetes finalizers: deletion is a protocol, not a cleanup callback

First principles

Kubernetes deletion is often asynchronous because an API object may represent something outside the API server. A custom resource can describe a cloud database, a DNS record, or a storage allocation. Removing only the Kubernetes record would leave that external resource behind. A finalizer gives a controller time to reconcile the external effect before the API object disappears.

When a client requests deletion, the API server typically sets metadata.deletionTimestamp and returns the object while its metadata.finalizers list is nonempty. The object is now terminating. A controller watches it, performs or confirms cleanup, and removes the specific finalizer string it owns. Once no finalizers remain, the API server can finish deleting the object. A finalizer is a coordination marker, not executable code: the controller must be running and must implement the cleanup path.

This creates a small distributed protocol. A delete request can be retried, a controller can restart between cleanup and finalizer removal, and the external provider can time out after performing the deletion. Therefore cleanup should be idempotent. “Resource not found” may mean cleanup already succeeded. A transient provider error should keep the finalizer and be retried with bounded backoff. Removing the finalizer after an ambiguous network error can orphan the external resource; retaining it forever can block namespace deletion and operations.

Finalizers differ from owner references and garbage collection. An owner reference expresses Kubernetes object ownership and may trigger dependent-object deletion. A finalizer gives a controller a chance to coordinate work before the owning object is removed. Neither mechanism guarantees that a cloud API is reachable or that the controller has credentials. The controller must make those operational dependencies observable.

Runnable example

Use this read-only command to inspect why a custom object is still terminating. Replace the resource type and name with values from your cluster. The output shows the timestamp and finalizer strings; it does not identify which controller is healthy or prove that external cleanup is safe to skip.

kubectl get widgets.example.net sample -n demo -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}'

An output like 2026-10-06T12:30:00Z followed by [example.net/cleanup] means deletion was requested and at least one finalizer remains. Next inspect the controller deployment, its events and logs, and the external resource using its stable provider-side identifier. Confirm whether cleanup completed before taking any action. Do not patch the finalizer list just to make the kubectl delete command appear successful.

In controller code, use a narrow ownership rule: add your finalizer before creating an external resource, reconcile deletion whenever deletionTimestamp is set, and remove only your own finalizer after confirmed cleanup. The resource identifier should be persisted in the custom object status or another durable mapping so a restarted controller can find what it owns. If creation succeeded remotely but the response was lost before status was written, the controller needs a deterministic external name or discovery mechanism to avoid creating duplicates.

Contextual failure case

A platform team removes a namespace while a cloud provider API is unavailable. Several custom resources remain terminating because their finalizers cannot complete cleanup. An operator deletes the finalizer strings manually, the namespace disappears, and the cloud resources continue billing. The visible Kubernetes symptom is resolved, but ownership information and the controller's retry opportunity have been destroyed.

There are cases where manual removal is appropriate, such as a permanently removed controller with a documented orphan-recovery process. The safe action is not “clear all finalizers”; it is to establish the external resource state, assign a new owner for cleanup or accept and record the orphan, and remove only the known stale finalizer. Preserve object snapshots and external identifiers before intervention. A bulk patch can bypass unrelated controllers' cleanup contracts.

A different bug occurs when a controller adds its finalizer after it creates the external resource. If the controller crashes in between, deletion can race ahead with no marker requesting cleanup. Add the finalizer first, then create external state, and make retries safe. This order may temporarily leave a finalizer when creation fails, but that is recoverable: the controller can observe that no external resource exists and remove its marker.

Verification

Build a test that creates a custom resource, observes the finalizer, requests deletion, and asserts the object remains until the fake provider confirms cleanup. Simulate a timeout after the provider has already deleted the resource; a retry should treat “not found” as success and remove the finalizer. Also test controller restart between provider cleanup and finalizer removal. These tests exercise protocol boundaries that a happy-path unit test misses.

In a cluster, inspect object events, controller availability, reconciliation metrics, and provider request IDs. Alert on the age and count of terminating objects by finalizer key rather than only on delete command failures. Tie the diagnosis to the desired-state and controller model in Kubernetes reconciliation. Never expose cloud credentials in controller logs while debugging provider failures.

Exercise

Design a finalizer for a custom database resource. Define the ordering for adding the finalizer, creating the database, recording its external ID, handling a duplicate create response, deleting the external resource, and removing the finalizer. Then inject crashes after each step. For each restart, show how the next reconciliation distinguishes “not created,” “created but not recorded,” “deleted but not acknowledged,” and “fully cleaned up.”

Safety and caveats

Finalizer strings are controller-specific and have no universal cleanup semantics. A finalizer may remain because of a controller bug, unavailable credentials, API throttling, or a deliberate policy. Do not assume an object is safe to force-remove based on age alone. Cluster versions, CRD implementations, and provider behavior affect details; test recovery procedures in a disposable environment and retain an auditable record when accepting external orphans.