sh scripts/bench.sh baseline # run everything, save it as bench-out/baseline.txt
# ...change something...
sh scripts/bench.sh compare # run again and diff: exits 1 on a regression
compare prints a table of ns/op with the delta per benchmark plus allocation
deltas, and fails when any benchmark is worse than BENCH_THRESHOLD_NS
(default 25%; allocation growth over BENCH_THRESHOLD_ALLOC, default 40%, is a
warning). The verdict is machine-readable too:
python3 scripts/bench_compare.py --json bench-out/baseline.txt bench-out/compare-run.txt
Noise on a shared machine is real — single runs of the sub-millisecond
benchmarks swing ±30%. Use BENCH_COUNT=3 (or more) before trusting any delta
under 25%, keep the filter tight while iterating
(BENCH_FILTER=Render scripts/bench.sh compare), and when a change matters,
alternate the two code versions in the same session (a git stash of the four
hot files, then swap) as this pass did. BENCH_OUT=/tmp/b keeps results out
of the tree.
The baseline itself carries its provenance (date, host, Go version) as comment
lines, so an old baseline says what it was measured against. Keep it in the
worktree out of git (bench-out/ is ignored) — it is a local instrument, not
an artifact to share.