performance / the-workflow-baseline-change-compare

performance / the-workflow-baseline-change-compare

The workflow: baseline, change, compare

v0.12.0 · 05 / 06 · 1 min read · docs/BENCHMARKS.md

The suite is built for before/after. scripts/bench.sh wraps it:

sh scripts/bench.sh baseline     # run everything, save it as bench-out/baseline.txt
# ...change something...
sh scripts/bench.sh compare     # run again and diff: exits 1 on a regression

compare prints a table of ns/op with the delta per benchmark plus allocation deltas, and fails when any benchmark is worse than BENCH_THRESHOLD_NS (default 25%; allocation growth over BENCH_THRESHOLD_ALLOC, default 40%, is a warning). The verdict is machine-readable too:

python3 scripts/bench_compare.py --json bench-out/baseline.txt bench-out/compare-run.txt

Noise on a shared machine is real — single runs of the sub-millisecond benchmarks swing ±30%. Use BENCH_COUNT=3 (or more) before trusting any delta under 25%, keep the filter tight while iterating (BENCH_FILTER=Render scripts/bench.sh compare), and when a change matters, alternate the two code versions in the same session (a git stash of the four hot files, then swap) as this pass did. BENCH_OUT=/tmp/b keeps results out of the tree.

The baseline itself carries its provenance (date, host, Go version) as comment lines, so an old baseline says what it was measured against. Keep it in the worktree out of git (bench-out/ is ignored) — it is a local instrument, not an artifact to share.

edit this page on GitHub ↗report a problem ↗