Establish performance baselines and regression detection

[dheerajodha]

What:

Store benchmark baselines in the repo and compare CI results against them. Fail the benchmark check when regressions exceed configurable thresholds.

Why:

EC-1818 added a report-only stress benchmark to CI. This follow-up closes the loop by detecting regressions automatically, without baselines and thresholds, the benchmark runs but nobody notices when performance degrades.

Tickets:

EC-1819

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 1a8ce0d0-7184-434b-86b1-92630543a50f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 1:06 PM UTC · Completed 1:25 PM UTC
Commit: 87c4a29 · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
acceptance	54.31% <ø> (+0.05%)	⬆️
generative	16.78% <ø> (-0.02%)	⬇️
integration	27.94% <ø> (-0.03%)	⬇️
unit	71.76% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fullsend-ai-review]

Review

Findings

Medium

• [logic-error] Makefile:209 — The benchmark_baseline target defaults EC_STRESS_WORKERS to 35 (${EC_STRESS_WORKERS:-35}), while the CI workflow hardcodes EC_STRESS_WORKERS: "10" and the checked-in baseline.json records "workers": 10. Running make benchmark_baseline without explicitly setting EC_STRESS_WORKERS=10 generates a baseline with 35 workers that CI will compare against using 10 workers, producing an unfair comparison that could mask regressions or trigger false positives.
  Remediation: Change ${EC_STRESS_WORKERS:-35} to ${EC_STRESS_WORKERS:-10} in the Makefile to match CI.

• [edge-case] benchmark/stress/compare.sh:55 — current_ns and current_rss are extracted via grep -oP with no validation that they are non-empty. If the benchmark output format changes and metrics are missing, grep produces empty strings and awk treats them as 0, causing a misleading −100% regression that fails CI. The same gap exists in the workflow's "Write job summary" step where $peak_rss and $ns_op are used without fallbacks in the rss_change/time_change calculations.
  Remediation: Add guards after extraction: if [[ -z "$current_ns" || -z "$current_rss" ]]; then echo "ERROR: Could not parse benchmark metrics."; exit 1; fi

• [protected-path] .github/workflows/benchmark.yaml — This PR adds a file under .github/, which is a protected path requiring human approval. The PR body provides rationale and references Jira ticket EC-1819. Human review of this CI workflow change is required regardless of automated review outcome.

Low

• [pattern-inconsistency] benchmark/stress/compare.sh:56 — compare.sh uses python3 for JSON parsing in 4 places, and the workflow uses it in 2 more. The rest of the codebase uses jq. Consider using jq for consistency, e.g.: baseline_ns=$(jq -r '.execution_time_ns' "${BASELINE}")

• [error-handling-idiom] Makefile:203 — The benchmark_baseline target suppresses stderr (2>/dev/null). If the benchmark panics or encounters build errors, no diagnostics are shown.

• [pattern-inconsistency] .github/workflows/benchmark.yaml:21 — Existing workflows (checks-codecov, lint, codeql) trigger on both pull_request and push. This workflow omits push, so merges to main won't produce benchmark runs. If intentional, a comment explaining the rationale would help.

• [edge-case] benchmark/stress/compare.sh:79 — The threshold comparison (change > thresh) only detects positive regressions. A dramatic improvement (e.g., −90%) passes silently, which could indicate a broken benchmark rather than genuine improvement.

• [code-organization] Makefile:200 — The benchmark_baseline target name matches the existing benchmark_% pattern rule. GNU Make prefers the explicit rule, but the overlap may confuse readers. (Note: benchmark_data and benchmark_simple follow the same pattern, so this is consistent with existing convention.)

• [missing-doc] benchmark/README.md — The benchmark README doesn't mention baseline generation, make benchmark_baseline, compare.sh, or threshold configuration.

• [missing-doc] AGENTS.md — AGENTS.md documents build/test commands but doesn't list benchmark make targets (pre-existing gap widened by this PR).