Concept notes · Signals

Measuring load and health

A measurement answers only the question it asks. Know what a load test offers, what a ratio counts and what a profile samples.

Open and closed load

A load test is an experiment: you offer work at some rate and measure what completes. How the generator decides when to send matters as much as how much it sends.

In a closed model a fixed number of virtual users each send a request, wait for the answer, then send the next. The offered rate is users ÷ (response time + pause), so when the system slows down the generator slows with it. In an open model requests arrive on a schedule, whatever happens to earlier ones, the way independent customers do. When the system slows down, work piles up. In k6, constant-vus and ramping-vus are closed; constant-arrival-rate and ramping-arrival-rate are open.

How it fails. A closed test against a stalling service reports calm error rates and simply fewer requests: the demand that would have arrived during the stall was never sent. Results that count only completed requests hide the demand left waiting. Illustrative: against a service that completes about five requests a second, four closed users finished 53 requests in ten seconds with no failures. An open schedule of 20 a second, with at most four users to send them, finished the same 53, and k6 also reported 148 scheduled iterations it could not start. An open test is only as good as its generator: when no user is free, or the generator runs short of CPU, the schedule slips.

Observe it.

export const options = { scenarios: { orders: {
  executor: 'constant-arrival-rate', rate: 20, timeUnit: '1s',
  duration: '10m', preAllocatedVUs: 50, maxVUs: 200 } } };

An open scenario in a k6 script: 20 iterations started every second, with up to 200 users available to start them.

k6 run --summary-trend-stats 'avg,p(95),p(99),max' script.js

Read iterations, http_reqs, failures and dropped_iterations: scheduled arrivals that never started. Choose the percentiles your target is written in.

top -b -d 1 -n 10 -p "$(pgrep -x k6)"

Watch the generator during the run. A generator near a full core cannot keep a schedule.

What good looks like. The offered rate is stated and achieved, with dropped_iterations at zero; completed, failed and dropped work add up to every scheduled arrival; and the generator has headroom throughout.

A success ratio and the rule that pages

A service level indicator (SLI) is a ratio: good events divided by valid events over a window. The objective (SLO) is a target for that ratio over a longer period, for example 99.5% over 28 days, and the error budget is what the target leaves: 0.5%. Decide first which events are valid, then which of those count as good. From Prometheus counters, a request success ratio is:

sum(rate(http_requests_total{code!~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Successful request rate over all request rate, both over five minutes. rate handles counter resets.

An alerting rule evaluates an expression on a schedule. Every series the expression returns is an alert; it is pending until it has been returned continuously for the rule’s for duration, then firing, and Alertmanager routes it by its labels.

How it fails. A rule fires only on series that exist. If the expression returns nothing, nothing fires, whatever the reason. With no traffic the ratio is 0 ÷ 0, which is NaN, and NaN < 0.99 is false. A long for or a long window smooths short outages away.

Observe it.

curl -sS -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:request_success:ratio_rate5m'

Evaluate the SLI as the rule sees it now, here through a recording rule that stores the ratio above under that name. An empty result means there is nothing to alert on.

curl -sS http://localhost:9090/api/v1/rules
curl -sS http://localhost:9090/api/v1/alerts

Every loaded rule with its health, last error and state; then the alerts currently pending or firing.

promtool check rules rules.yml
promtool test rules tests.yml

Check syntax, then run unit tests that feed synthetic series through the rules and state which alerts must fire.

What good looks like. The SLI returns a value for every service and window you care about, a separate rule using absent() covers “no data”, and a test has shown the alert going from pending to firing and reaching its receiver.

Ask the running process where its time goes

A sampling profiler interrupts a running process many times a second and records the stack it was executing. Functions that appear in many samples are where the CPU time went. That is an on-CPU profile: time spent waiting on locks, disks or the network appears only in off-CPU or wall-clock views. Linux perf samples any process. Managed runtimes add their own recorders: the JVM’s Flight Recorder (JFR) samples methods and allocations and records garbage collection and lock events, at low enough overhead to use in production.

How it fails. A profile answers the question you sampled. A recording of the wrong process, of a quiet minute, or too short to collect many samples shows nothing useful. A process that is slow because it waits looks idle in a CPU profile. Frames from code compiled at run time can appear as bare addresses unless the profiler has their symbols.

Observe it.

top -H -p 4242

CPU per thread of process 4242. First decide whether the process is busy at all, and in which threads.

perf top -p 4242
perf record -F 99 -g -p 4242 -- sleep 30
perf report --stdio

perf top shows the hottest functions live. perf record samples at 99 Hz with call stacks for 30 seconds into perf.data, which perf report summarises.

jcmd 4242 JFR.start duration=60s filename=/tmp/app.jfr
jfr summary /tmp/app.jfr
jfr print --events jdk.ExecutionSample /tmp/app.jfr

For a JVM: record for 60 seconds, count the events of each type, then print the sampled stacks. jcmd 4242 help lists the other diagnostic commands, such as GC.heap_info and Thread.print.

What good looks like. The profile was taken while the symptom was present and is compared with one from a healthy period. The hottest stacks name code you can point to, and the CPU they use explains the latency, or the profile shows clearly that the time is spent waiting.

References