Concept notes · Distributed

Agreement and acknowledgement

How replicated systems decide that a write happened, who may act, and when a commit is allowed to return.

Commands use etcd’s etcdctl (API v3) and PostgreSQL’s psql. Add --endpoints and the TLS flags (--cacert, --cert, --key) your cluster needs.

A write needs a majority

Consensus systems such as etcd (Raft) accept a write only after a majority of voting members have stored it. The quorum for n voters is ⌊n/2⌋ + 1: two of three, three of five. A cluster keeps accepting writes while no more than n − quorum voters are lost: one of three, two of five. Four voters need three and still survive only one loss, so even sizes add cost without adding tolerance. Non-voting members, called learners in etcd, do not count.

Any two majorities share at least one member, and that member has seen every committed write. That overlap is why a minority cut off from the rest cannot accept writes: it could not know whether it conflicts with what the majority did.

How it fails. When fewer than a quorum of voters can reach each other, writes stop: requests time out or report that there is no leader. Every surviving member can still be up and answering health questions about itself. Reads may still be served, and may be stale, depending on how the client asks. Members already down count against the margin before planned work takes another one away.

Observe it.

etcdctl member list -w table

Every member, its status and IS LEARNER. Count the voters and work out the quorum.

etcdctl endpoint status --cluster -w table

Per member: IS LEADER, RAFT TERM and RAFT INDEX. Members should agree on one leader and one term; an index far behind the others marks a member that is not keeping up.

etcdctl endpoint health --cluster

Each member answers a read that needs a quorum, and the time it took or the error is reported. Nothing is written, although the output says “committed proposal”. From etcd 3.5 a member with an active alarm, such as NOSPACE, is also reported unhealthy; etcd 3.4 still calls it healthy, so check etcdctl alarm list as well.

What good looks like. All voters agree on the leader and term, their indexes are close, and the number of healthy voters exceeds the quorum by at least the number you may lose during planned work.

One leader, by lease

When only one replica may do a piece of work, such as running a schedule, the replicas elect a leader through a shared record with an expiry: a lease. In Kubernetes it is a Lease object (coordination.k8s.io/v1) holding holderIdentity, leaseDurationSeconds, acquireTime, renewTime and leaseTransitions.

Every candidate reads the same Lease. If it has no holder, or the holder’s renewTime plus leaseDurationSeconds has passed, the candidate writes itself in as holder. The API server accepts only one write against a given resourceVersion, so only one candidate wins. The leader renews well inside the duration; the others keep checking and take over only after it stops. A leader that cannot renew must stop acting before its lease runs out, because another replica may hold it by then.

How it fails. Leader-only work happens twice, from two replicas that both log that they lead, or not at all, while a holder that has gone away keeps the lease until it expires. A pause long enough to miss renewals, such as a stalled process, can leave a former leader acting after its lease has passed to another.

Observe it.

kubectl get leases -n sample-app

One row per lease with its HOLDER. Each election should appear once.

kubectl get lease report-scheduler -n sample-app -o yaml

Read holderIdentity, renewTime, leaseDurationSeconds and leaseTransitions. Run it twice a few seconds apart: renewTime should move.

kubectl logs -n sample-app -l app=report-scheduler --since=10m

Find which replicas say they acquired or lost leadership, and when, then compare with the Lease’s holder.

What good looks like. Exactly one holder per election, renewTime advancing, transitions only on real handovers, and leader-only effects appearing once per period, from the holder.

When a consensus store runs out of room

Each etcd member keeps a complete copy of the data in one backend file. The server enforces a size limit on it, --quota-backend-bytes, which defaults to 2 GiB; many distributions set more. When a member’s backend would grow past the quota, etcd raises a NOSPACE alarm for the whole cluster.

While that alarm is active the cluster is in a maintenance mode: it serves reads and accepts deletes, and refuses writes that add data with etcdserver: mvcc: database space exceeded. The alarm does not clear by itself. Once space has really been reclaimed, an operator disarms it with etcdctl alarm disarm; disarming while the backend is still over the quota raises it again on the next write.

How it fails. Everything that only reads keeps working, and every change fails. In front of a Kubernetes API server that means listing and describing work while creates, updates, status changes and lease renewals all fail together.

Observe it.

etcdctl alarm list

An active alarm prints as memberID:10276657743932975437 alarm:NOSPACE. No output means no alarm.

etcdctl endpoint status --cluster -w table

DB SIZE per member, compared with the configured quota; the ERRORS column repeats any alarm.

etcdctl endpoint status -w json

The same status in bytes: dbSize is the size of the backend file, and dbSizeInUse is the part of it holding data.

What good looks like. No alarms, DB SIZE well below the quota on every member, and test writes succeeding. Growth is watched as a trend, so the quota is never the first warning.

A commit can wait for a standby

A PostgreSQL primary streams its write-ahead log (WAL) to standbys. With asynchronous replication, the default, COMMIT returns once the WAL is flushed locally; a standby may lag, and a failover can lose the newest commits. With synchronous replication, COMMIT also waits for confirmation from standbys named in synchronous_standby_names, for example FIRST 1 (replica_a, replica_b) or ANY 2 (replica_a, replica_b, replica_c).

A standby is known by the application_name it connects with, normally set in its primary_conninfo. synchronous_commit sets how far the confirmation goes: remote_write (received and written), on (flushed to the standby’s disk) or remote_apply (replayed and visible to its queries). local and off do not wait for a standby at all.

How it fails. When the required standbys do not confirm, commits wait, with no timeout. The transaction is already durable on the primary, but the client gets no answer; reads and read-only work carry on. Cancelling the wait does not undo the commit: PostgreSQL warns canceling wait for synchronous replication due to user request and adds that the transaction has already committed locally.

Observe it.

psql -c "SHOW synchronous_standby_names" -c "SHOW synchronous_commit"

The standbys commits wait for, and how far each confirmation must go. An empty name list means commits wait for no standby.

psql -c "SELECT application_name, state, sync_state, replay_lag FROM pg_stat_replication"

One row per connected standby. sync_state is sync, quorum, potential or async.

psql -c "SELECT pid, wait_event_type, wait_event, state, query FROM pg_stat_activity WHERE wait_event = 'SyncRep'"

Sessions waiting for standby confirmation show wait_event_type IPC and wait_event SyncRep.

What good looks like. As many standbys as the setting requires show sync or quorum and are streaming, commits return in normal time, and no session waits on SyncRep.

References