Distributed Systems Foundations: Consistency, Retries, and Delivery Semantics

Partial failure is normal

In a distributed system, a caller can lose its connection after the server committed a write but before the response arrived. A broker can acknowledge a message while a consumer crashes before recording its offset. A replica can be reachable but behind. These are ordinary uncertainties created when independent machines communicate over networks.

A timeout says the caller did not receive a response within its budget. It does not prove that the operation did not happen. Retrying may be required, but repeating a non-idempotent effect can charge twice, send duplicate notifications, or create conflicting records.

Consistency describes observable histories

Consistency guarantees describe what values clients can observe after operations. Stronger guarantees can make reasoning easier but may require coordination and reduce availability during partitions. Weaker or eventual guarantees can improve latency or availability, but clients and applications must tolerate stale reads, convergence delay, or conflicts.

Avoid describing a system simply as “consistent” without naming the guarantee and scope. Is read-after-write guaranteed for one client? Can two users see different values temporarily? Does a successful response mean the data is durable on one node or replicated to a quorum? Concrete answers expose design trade-offs.

Delivery and processing are separate

At-most-once processing may lose work but avoids repeated processing. At-least-once delivery retries until acknowledged, which means duplicates are possible. “Exactly once” claims need careful scope: a broker may deduplicate a publish within a defined transaction, but an external side effect such as sending email still needs an idempotency strategy.

A common robust design uses at-least-once delivery and makes the business effect idempotent. Store a stable operation or event ID under a uniqueness constraint, apply the state transition once, and return the previously recorded outcome on retry. Bound retry attempts and use backoff with jitter so many clients do not synchronize into a retry storm.

Make failure policy explicit

Define deadlines across service calls, distinguish retryable errors from permanent validation failures, and decide how poison messages are isolated. Measure queue age and retry counts, not just consumer process health. If the business requires ordering, state the scope, often per account or aggregate, rather than assuming global ordering.

A usable mental model

Ask what a caller knows after each timeout, what a reader can observe during replication, and whether repeating each effect is safe. These questions help choose storage consistency, message acknowledgement, idempotency keys, and recovery procedures without relying on vague exactly-once promises.