06 · Investigation
Incident investigation
Replace “try something” with a prediction you can check. Keep the result even when your theory is wrong.
- 2-minute guide
- All field guides
The model
First establish impact and consider a safe, reversible mitigation. Then locate the failing boundary. Write one hypothesis, the observation that would challenge it, and a fixed test window. Change one thing and compare.
A recent change is a lead, not proof. A missing log line may reflect sampling or incomplete collection, not a missing request. Record what each observation actually establishes.
Worked example
Illustrative: a worker pool has 120 queued jobs. You reduce arrivals to 80 jobs/s, below its measured capacity of 100 jobs/s. If both rates hold, spare capacity is 20 jobs/s, so the queue should drain in 6 seconds.
Watch for 30 seconds. If the queue stays at 120, verify that arrivals really fell and workers still complete 100/s. The prediction depended on both. An unchanged error rate alone does not disprove overload; your intervention may have been insufficient.
Commands worth running
Keep observations, changes and outcomes together in a timestamped note. Kubernetes examples use a fictional worker deployment.
date -u '+%H:%M:%S UTC'Stamp the observation before interpreting it. Use one timezone across your notes and dashboards.
kubectl events --types=WarningInspect warnings in the current namespace. Events are clues, not a complete change history; a quiet list does not prove nothing changed.
kubectl logs deployment/worker \
--since=5m --timestampsCompare one pod’s request timings with your test window. Widen collection when other replicas matter; missing evidence is not negative proof.
Self-check
The queue did not shrink. What can you conclude?
Show answer
The prediction or its conditions did not hold. Check the two measured rates before declaring the hypothesis disproved. Save that narrower conclusion for the next investigator.