Searching lab logs

The model

Every line a lab service writes is stored as an event with its source (narrow to one with source=<name>), its time and any fields found in it: JSON keys, key=value pairs and access-log columns such as status and path. A search is a filter, then a pipeline: each | hands the events to the next command.

Searches only ever see your session. The search box starts with index=<your storyline>; keep that token in the examples below and replace the placeholder with the storyline already shown in your box. Other indexes are not available here. Leave out session=. Lines arrive within seconds.

Worked example

Checkouts fail. Chart errors by source, then group them, then pull a number out of the text:

index=<your storyline> status>=500 | timechart span=1m count by source

index=<your storyline> error OR timeout | stats count by source

index=<your storyline> * | rex "took=(?<took>\d+)ms" | where took>500 | table _time source took

Groups leave out events that lack the field, so a missing status never shows as an empty row.

Commands worth running

index=<your storyline> status>=500 | head 20

index=<your storyline> * | stats count by source

index=<your storyline> * | top limit=5 path

Supported here:

• where stats eval rex regex spath
• sort head tail dedup fields rename fillnull
• top rare timechart bin (bucket) streamstats makemv mvexpand
• chart eventstats transaction (last)
• reverse transpose addtotals uniq replace convert strcat delta fieldsummary (then sort/where/table)

New fields: add | table.

Not here: subsearches, join, append, lookup.

Self-check

• Why can a search show fewer events than you expect?

  The time range (30 second window holds only the last 30 s), or a filter on a field some lines do not carry.

References

• Google SRE: monitoring distributed systems
• Kubernetes logging architecture
• journalctl manual

Practise it

Practise: any lab on the board