Consensus and Observability Advanced: Quorums, Split Brain, and Trace Context

1. Quorum is the safety boundary

Raft and ZooKeeper-style systems use majorities to prevent two partitions from independently committing conflicting histories. In a five-voter group, a quorum is three. A 3/2 split allows the three-node side to elect or retain a leader; the two-node minority cannot safely commit. If no partition can reach three, writes stop rather than allowing competing leaders.

This is the safety/availability trade: during a partition, a system that preserves a single authoritative history may reject writes on the minority side. More replicas do not remove the quorum rule. Understand which members vote, which are learners, and whether the deployment's failure domains actually match the assumed partition model.

2. Trace context is an explicit carrier

A trace usually has a trace ID and parent/span ID. W3C Trace Context encodes this in traceparent; OpenTelemetry instrumentation injects it into HTTP headers and should also inject/extract it in Kafka record headers. A broker is an asynchronous boundary: consumer execution may happen on another process hours later, so thread-local context alone is not enough.

Baggage carries small cross-service attributes, not secrets or large payloads. When work enters a thread pool, verify the context propagator installs and clears context per task. MDC-based logging must also be cleared after execution to prevent request IDs leaking between reused worker threads.

3. Exactly-once boundaries

Exactly-once processing is always scoped. Kafka transactions can atomically publish records and commit consumed offsets within Kafka, but a database write or non-idempotent HTTP call remains outside that transaction. Use an outbox/inbox, stable event IDs, deduplication constraints, and idempotent business transitions to make retries safe across system boundaries.

Event sourcing stores domain events; CQRS creates read models optimized for queries. Those projections are often eventually consistent. Expose the expected read-your-write behavior and support rebuilding projections from versioned, replay-safe events. Snapshotting reduces replay cost but snapshots need schema/version controls and verification.

Interview and troubleshooting drills

Interview: In a five-node consensus group during a 3/2 network partition, which side can commit and why? Explain majority intersection and why the minority must stop accepting authoritative writes.

Troubleshooting: A trace disappears between an HTTP handler and Kafka consumer. Verify producer injection into message headers, consumer extraction, instrumentation versions, and context restoration across worker pools. Carry traceparent as message metadata; do not assume thread-local MDC crosses the broker.

Challenge: Explain why a broker's exactly-once mode cannot make a downstream payment API execute once. Identify the local transaction boundary, external uncertainty after timeout, and idempotency key or outbox mechanism that makes the overall workflow recoverable.