Fencing tokens: reject workers whose lease has expired

First principles

A distributed lease grants authority for a bounded interval. A worker can lose that authority while remaining alive: its process pauses, its network connection stalls, or the host stops scheduling it for several seconds. Another worker acquires the expired lease and begins useful work. When the original worker resumes, both can believe they should write. Checking lease ownership before a write leaves a gap between the check and the effect, so it cannot close this race on its own.

A fencing token is a monotonically increasing generation issued with ownership. The first owner might receive generation seven and its replacement generation eight. The protected resource remembers the highest generation it has accepted. Once it accepts eight, it rejects writes carrying seven. Authority is checked where the effect happens, rather than relying on every client to stop behaving correctly after expiry. The resource becomes the final enforcement point for stale ownership.

This mechanism has a precise limit. If an old worker reaches the destination before any newer token does, the destination may still accept it. A fence does not consult lease time or detect expiry by magic; it rejects generations known to be superseded. Some systems need stronger coordination with the lease authority or a resource-specific ownership protocol. State the required invariant first: preventing an older generation from overwriting an accepted newer generation is different from proving every write occurred during a valid lease.

Runnable example

Run this Python 3 program locally. It models destination-side checking in one process, using a lock to keep comparison and mutation indivisible. The numbers are supplied by the test; production tokens need a durable authority that orders ownership changes across failures.

from threading import Lock

class FencedRegister:
    def __init__(self):
        self.highest = 0
        self.value = None
        self.lock = Lock()

    def write(self, token, value):
        if not isinstance(token, int) or token <= 0:
            raise ValueError("positive integer token required")
        with self.lock:
            if token < self.highest:
                raise RuntimeError("stale owner")
            self.highest = token
            self.value = value

register = FencedRegister()
register.write(7, "old worker started")
register.write(8, "replacement completed")
try:
    register.write(7, "old worker resumed")
    raise AssertionError("stale write accepted")
except RuntimeError as error:
    assert str(error) == "stale owner"
assert register.value == "replacement completed"
register.write(8, "same owner updated")
print(register.highest, register.value)

Allowing equal tokens permits several writes by one owner. That is a design choice, not automatic duplicate suppression. Two requests with the same token can still reorder or repeat effects. If the owner writes a sequence of state transitions, combine the fence with per-operation identity or an expected revision. A fencing token identifies an ownership generation; an idempotency key identifies a logical operation. Using one field for both can accidentally reject valid owner updates or allow duplicate operations.

Contextual failure case

A reconciliation worker holds a lease while computing account totals. It pauses after reading old inputs. The lease expires, a replacement computes current totals, and the destination accepts the replacement's write. The paused worker resumes and overwrites the total with stale data. Increasing the lease duration reduces one probability but leaves the mechanism intact. Adding a fencing check at the destination rejects the old generation even when the process cannot be stopped.

Now imagine the destination checks the token with a read, then performs an unconditional update. Two requests interleave between those steps, allowing the older generation through after the newer write. The comparison and mutation must share one atomic boundary, such as a conditional database update under appropriate concurrency control. Similarly, protecting a metadata row while writing an unfenced remote object does not protect that object. Every consequential effect needs an enforcement path or a reconciliation strategy.

Verification

The example prints 8 same owner updated. Add cases for zero tokens, a newer generation, and a same-generation update. Also test the limitation explicitly: a fresh register accepts seven until it has observed eight. Use payload inspection to design a synthetic request containing resource identity, ownership generation, operation identity, and expected revision. Keep those fields distinct and specify which service validates each one.

In a database implementation, run two independent clients and force the older client to pause after reading. Resume it only after the newer client's conditional write commits. The old write should affect zero rows or return a documented conflict. A single-threaded unit test of a read-then-write implementation would not expose that race, so include the interleaving that violates the invariant.

Exercise

Extend the register with an operation ID and a stored result. A repeated operation in the same generation should return its previous result rather than apply twice. A different operation in the same generation should remain valid. An operation from an older generation should fail even if its ID is new. Describe the retention policy required for remembering operation IDs and what happens when a delayed retry arrives after that retention window.

Safety and caveats

Never derive authoritative generations from unsynchronized wall clocks or reset a local counter after restart. Authentication must prevent callers from inventing an arbitrarily high token that fences everyone else out. Persist highest accepted generations with the protected state; losing that record can reauthorize stale workers. Tokens do not provide consensus, transactional delivery, or permission checks by themselves. Use them as one explicit defense at a resource boundary, and monitor rejected writes as evidence of ownership overlap rather than dismissing them as harmless noise.