02 · Latency & retries
Every attempt is load
A timeout means the caller stopped waiting. The dependency may still be working—or may already have committed the operation.
- 2-minute guide
- All field guides
The model
First attempts + retries = downstream work. If a retry overlaps its unfinished predecessor, one user request occupies multiple workers. More waiting can trigger more retries: a feedback loop.
Bound attempts within one overall deadline. Backoff and jitter spread retries out; they do not make unsafe side effects repeatable. Retried writes need an idempotency mechanism.
Worked example
Search receives 100 requests/s. Its retry counter shows another 50 attempts/s, so the ranker receives 150/s: 1.5 attempts per request. At a measured capacity of 120/s, queued work grows by 30/s.
Here the retry counter explains the increase. A higher dependency rate alone could instead mean fan-out, another caller, or a changed cache hit rate. Attribute the extra work before choosing a change.
Commands worth running
Use fictional service names below as placeholders, in the caller’s namespace.
kubectl logs deployment/search \
--since=5m --timestampsTake one failed request ID and its deadline from the caller’s incident window. Deployment logs show one pod, not the whole fleet.
kubectl logs deployment/ranker \
--since=5m --timestampsFind that ID downstream. Later completion can explain a caller timeout; repeated arrivals support the retry hypothesis.
curl -sS --max-time 0.25 -o /dev/null \
-w 'HTTP %{http_code}\n' http://ranker:8080/rankCap the whole transfer at 250 ms. Exit 28 means curl timed out; it does not establish that server work stopped.
Self-check
What must you check before retrying a timed-out write?
Show answer
Whether it already took effect, and whether repeating it is idempotent. A missing acknowledgment is not proof of a missing write.