05 · Change safety

Safe rollouts and recovery

A rollout tracks deployment progress. Recovery means real operations work again. Verify both.

The model

Expose a small canary group to a change while keeping a control group unchanged. Compare similar request mixes over the same window; an absolute error threshold can miss a regression.

Know the reversal boundary first. A Deployment revision contains its Pod template. Rolling it back does not restore a ConfigMap edited in place, a Service, or database data. Check compatibility before reversing a release.

Compare rates for canary and controlComparable requests go to the unchanged control and the new canary. In one thousand requests each, the control has five failures and the canary fifty: one-half percent versus five percent, a tenfold difference. Comparable request mix Control5 / 1,000 fail0.5% Canary50 / 1,000 fail5% Canary error rate: 10× control
Illustrative — equal sample sizes, same observation window.

Worked example

In 1,000 comparable requests per group, control has 5 failures and canary has 50. That is 0.5% versus 5%: ten times the error rate. Pause expansion and inspect the change, even if every pod reports Ready.

This comparison supports a release-related regression, not a complete diagnosis. Shared dependencies can affect both groups. After a bounded mitigation, repeat a real operation and watch sustained success and latency—not just rollout status.

Commands worth running

Example resource names are fictional. Inspect the intended namespace before acting.

kubectl rollout history \
  deployment/ledger --revision=11

Read a specific revision’s template. Do not assume “previous” means the known-good version after several releases.

kubectl get configmap ledger-config -o yaml

Inspect separately versioned configuration. A template referencing this name does not freeze its contents.

kubectl rollout status \
  deployment/ledger --timeout=60s

Bound the status watch to a minute. Success establishes rollout progress, not successful user operations; timeout does not cancel the rollout.

Self-check

References

Practise it

Costume availability is unknownPhoto estimates never arrive