01 · Signals
Reading service signals
A faster response is not always a better response. Read latency alongside what happened to the request.
- 2-minute guide
- All field guides
The model
Read four signals together: traffic measures demand, errors measure unsuccessful outcomes, latency measures elapsed time, and saturation measures work waiting for a constrained resource. They locate a question; none identifies a cause alone.
Keep the request rate beside every percentage. Zero errors with zero requests is unknown health, not success. Split latency by outcome before celebrating an improvement.
Worked example
Previously, all 100 requests finished successfully in 200 ms. Now 60 fail in 10 ms; 40 succeed in 200 ms. Success falls to 40%, while the median drops to 10 ms. The apparent speedup comes from failures. Investigate why requests are rejected, rather than tuning the successful path.
Commands worth running
These examples assume a Prometheus job named api and a classic latency histogram; substitute your metric names.
sum(rate(http_requests_total{job="api"}[5m]))Check demand over five minutes. Take rate before summing so counter resets remain visible.
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))Estimate tail latency in seconds. Preserve le when combining buckets; bucket width limits precision. Never average instance percentiles.
curl -sS --max-time 3 -o /dev/null \
-w 'code=%{http_code} total=%{time_total}s\n' https://api.example.test/healthzBound one diagnostic request and print its status and duration. A healthy health endpoint does not prove the real operation works.
Self-check
Which observation would challenge the “speedup” claim?
Show answer
The outcome split: 60% failed. Compare successful-request latency separately, then test the actual user operation.