07 · Distributed systems

Coordination and exactly-once

Each step reports its own success. Trace one effect across replicas, retries and handovers.

The model

Processes fail independently; messages are delayed, lost or repeated. Each step can only vouch for itself: a broker confirm, a finished Job, a Ready Pod. None of them proves that the business effect happened once.

Exactly-once effects need two pieces: at most one writer at a time (a lease, a lock, a quorum) and idempotent effects, where a repeat with the same key changes nothing. Each piece has a boundary: a lease expires, a quorum needs a majority, a shutdown has a deadline.

A retried payment applied onceCheckout sends payment k-41. The payments service commits it, but the reply is lost and checkout times out. Checkout retries with the same key; the service finds k-41 already committed and returns the first result. Two attempts produce one payment. Caller: checkout key k-41 payments commits k-41 Reply lostcaller times out Retry k-41first result
Illustrative — fictional names; two attempts, one payment.

Worked example

Of 1,000 payments, 20 time out after the server has already committed. The caller retries all 20. Without a key, 20 customers pay twice: 2%. With the key stored in the same commit as the payment, each retry returns the first result and the ledger holds exactly 1,000.

Ready is not agreement. Every replica can be healthy while two act as leader, or while too few remain for a majority. Health is per member; correctness is per group.

Commands worth running

Example names are fictional. Compare each result with the ledger, not the status.

kubectl get leases -o yaml

Read holderIdentity and renewTime. A holder that stops renewing for leaseDurationSeconds can be replaced; leadership is only as fresh as its last renewal.

kubectl get pdb -A

ALLOWED DISRUPTIONS is how many Pods voluntary evictions, such as drains, may remove right now. Direct deletes and crashes bypass the budget.

kubectl events --for job/settle-batch

Controllers record what they decided and when. Line those times up with the expected effect.

Self-check

References

Practise it

Claimants are being paid twiceEditors’ pages are being refusedRack maintenance takes the exchange offline