profiling: record deployment CPU time and peak memory per PR by leshy · Pull Request #3468 · dimensionalOS/dimos

leshy · 2026-08-14T16:37:26Z

No description provided.

bin/rusage runs a command in a transient systemd scope and reads the cgroup's own counters, so every child and native subprocess is counted -- /usr/bin/time sees neither unreaped grandchildren nor the tree's real memory high-water mark. Output is Bencher Metric Format, pushed from CI for a trend line. Record-only: no thresholds, so it cannot fail a build. --run-for gives a run a clean end. A replay stops producing but the blueprint keeps going, since from its side that is just a robot that went quiet, so bounded runs had no way to say when they were done.

checkout with lfs: true pulls everything and dies. get_data resolves data/.lfs/<name>.tar.gz for the one recording, in its own step so a pull failure is not mistaken for a profiling failure -- and so the download stays out of the measured window.

Compared against ci.yml's cockpit e2e job, which already runs a real deployment on the same replay dataset: it needs libturbojpeg for JPEG decode. Python is pinned because a moving interpreter would move the numbers this job exists to compare. The Bencher CLI rejects a project key passed as --token and wants BENCHER_API_KEY.

…test env --github-actions is the whole mechanism for a PR comment; without it Bencher stores the report and says nothing. Needs a pull_request trigger for PR context and pull-requests: write. Still no thresholds, so the comment is a table, never a failure. default-groups is ['tests'], so a plain uv sync dragged in pytest we never run. --extra unitree is what a go2 install actually pulls.

Bencher renders bytes as '3.06 units x 1e9'. MiB reads as a number a human can act on, and the measure now carries real units.

Cold start (--run-for 0) separates what the stack costs before it does any work from what the work costs -- core-seconds spent importing torch look identical to core-seconds spent on perception in the full number. Install size watches dependency bloat, which nothing else in CI does and which ships to the robot. push on this branch and pull_request both fired for the same commit, so every push profiled twice; concurrency cancels superseded runs.

A threshold is per measure, not per benchmark, so cold start and the full run shared one cpu-core-seconds limit sized for whichever is larger. It needs its own measures to be watchable; parking it until then. What it already told us: boot is 46% of CPU and 99% of peak memory.

--measure-prefix gives cold start its own measure names, so it lives under the same benchmark as the full run and still gets its own threshold -- the thing that made it unwatchable when it was a separate benchmark sharing cpu-core-seconds. Merge is a recursive jq merge, not add: both profiles write the same benchmark key, and add would silently keep only the last one's measures.

Bencher renders benchmarks as rows and measures as columns, so folding cold start in as coldstart-* measures made one row twenty columns wide. Its own benchmark is a second row instead. Separate benchmarks are fine under percentage thresholds, which compare each benchmark to its own history -- the objection only held for static limits. The user/sys split stays in the artifact and job summary but is filtered out of the Bencher push: it is what you read when a number moves, not something worth two columns on every comment.

Bencher's table is generated inside its Rust CLI and cannot be templated, so the layout now lives in bin/rusage-comment where it can be edited. Baseline comes from the metrics.json artifact of the last green run on main -- the same file this job already uploads, so no API is involved. Bencher's comment drops to --ci-only-on-alert: its wide table is worth reading exactly when something crossed a threshold.

main has never profiled, so every delta would render as '--' until this lands. Falling back to the branch's own last green run exercises the comparison now, and the table names the ref it used so the two cases are never confused.

The bash version buried the layout in a jq program. The template is now a plain string at the top of the file, and moves over 3% are bolded so a real change stands out from runner noise.

github-actions · 2026-08-14T18:07:09Z

Deployment resource footprint _{vs feat/ivan/rusage-profile}

unitree-go2-replay Wall 89.6s Δ+11% Cpu 107s Δ+8% Usr 73.2s Δ+20% Sys 34.3s Δ-10%

Mem 2.8GiB Δ+5% Thr 210 Δ+1% Rd 20MiB Δ+2258% Wr 82MiB Δ+264719% Rx 129.3GiB Δ+0% Tx 2.5GiB Δ+0%

venv Size 9.4GiB Δ-1%

The table cost a row per measure; one line per benchmark fits on a screen. Deltas are graded by magnitude through MathJax \textcolor, which is the only way to colour text in a GitHub comment without images. pids.peak comes from the same cgroup, so peak task count is free. Worth having here: the CLI already pins OMP_NUM_THREADS and KMP_BLOCKTIME because native pools sized themselves to all cores, and nothing else in CI would notice that coming back.

io.stat gives per-tree disk bytes; network has no cgroup file at all, so it comes from systemd's own eBPF IP counters, read from inside the scope while the unit is still alive. Both are optional: neither is available to an unprivileged user scope, and a measure the kernel did not report is omitted rather than sent as a zero that would read as real silence. Optional files are now copied out raw into a temp dir and parsed outside -- systemd expands ${...} in the command line before bash sees it, which was corrupting the in-scope arithmetic.

MathJax was the only way to colour a delta, and it dragged the serif math font in with it. Bold carries the emphasis instead and the comment keeps its own typeface. Values now carry their unit, rolling MiB up to GiB past 1024 so peak memory reads 2.7GiB rather than 2759.

Disk and network came back absent on the runner as well as locally, and a silently omitted measure looks identical to one that was never supported. The job now says which scope it used and whether io.stat and the IP counters were there.

Mem and Thr move to a second line so the first reads as time alone. The benchmark name links to its Bencher plots -- every number in the comment is one point on a trend line, and the trend is what is worth opening.

The diagnostic showed CI picking scope=user: runners do have a user manager, so the old probe never reached sudo. Only the system manager can enable the io controller or attach the eBPF counters behind IPAccounting, so a user scope silently loses both. Probing with sudo -n picks the system scope where it is free and falls back without ever blocking on a prompt. Verified locally: scope=system io.stat=yes ip=yes.

sudoers replaces PATH via secure_path even under sudo -E, so the payload lost uv the moment the scope switched to system. The runner did report io.stat=yes ip=yes on that run, so the switch itself was right.

codecov · 2026-08-14T18:49:14Z

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
4259	4	4255	69

View the top 3 failed test(s) by shortest run time

dimos.e2e_tests.test_dimsim_path_replaning::test_path_replanning

Stack Traces | 112s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fba7f7b90>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba0af6a0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e3fbab92990>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e40daa86750>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7e3fba52c0e0>

    @pytest.mark.self_hosted_large
    def test_path_replanning(
        lcm_spy, start_blueprint, dim_sim, direct_cmd_vel_explorer, spawn_wall_on_pose
    ) -> None:
        start_blueprint(
            "--dimsim-scene=empty",
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        # robot spawns at (3, 2)
    
        # side wall
        dim_sim.add_wall(2, -2.5, 12, -2.5)
        # other side wall
        dim_sim.add_wall(2, 3.5, 12, 3.5)
        # back wall (behind robot)
        dim_sim.add_wall(2, -2.5, 2, 3.5)
        # forward wall (far end)
        dim_sim.add_wall(12, -2.5, 12, 3.5)
        # dividing wall at x=7 with doors at y=[-1.5,-0.5] and y=[1.5,2.5]
        dim_sim.add_wall(7, -2.5, 7, -1.5)
        dim_sim.add_wall(7, -0.5, 7, 1.5)
        dim_sim.add_wall(7, 2.5, 7, 3.5)
    
        direct_cmd_vel_explorer.linear_speed = 0.8
>       direct_cmd_vel_explorer.follow_points([(10, 2), (2.5, 2), (3, 2)])

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e3fbab92990>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e40daa86750>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fba7f7b90>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7e3fba52c0e0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba0af6a0>

dimos/e2e_tests/test_dimsim_path_replaning.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e40daa86750>
        waypoints  = [(10, 2), (2.5, 2), (3, 2)]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e40daa86750>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e40daa86750>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

dimos.e2e_tests.test_manipulation_planning_groups::test_single_arm_plans_and_executes_through_control_coordinator

Stack Traces | 139s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e415007c080>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52dee0>

    def test_single_arm_plans_and_executes_through_control_coordinator(
        lcm_spy: LcmSpy,
        start_blueprint: Callable[..., DimosCliCall],
    ) -> None:
        """Plan with one arm and execute through its trajectory task."""
        _start_openarm_mock_planner(start_blueprint, lcm_spy)
    
        client = RPCClient(None, ManipulationModule)
        coordinator_client = RPCClient(None, ControlCoordinator)
        try:
>           [left] = _wait_for_groups(client, 1)

client     = <dimos.core.rpc_client.RPCClient object at 0x7e403e11a9c0>
coordinator_client = <dimos.core.rpc_client.RPCClient object at 0x7e403e19bb30>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e415007c080>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52dee0>

dimos/e2e_tests/test_manipulation_planning_groups.py:158: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <dimos.core.rpc_client.RPCClient object at 0x7e403e11a9c0>, count = 1

    def _wait_for_groups(
        client: RPCClient,
        count: int,
        *,
        timeout: float = 120.0,
    ) -> tuple[PlanningGroupInfo, ...]:
        deadline = time.time() + timeout
        last_error: BaseException | None = None
        while time.time() < deadline:
            try:
                groups = client.list_planning_groups()
                if len(groups) == count:
                    return groups
            except Exception as exc:
                last_error = exc
            time.sleep(0.5)
>       raise TimeoutError(f"Timed out waiting for {count} planning groups") from last_error
E       TimeoutError: Timed out waiting for 1 planning groups

client     = <dimos.core.rpc_client.RPCClient object at 0x7e403e11a9c0>
count      = 1
deadline   = 1786750358.870102
groups     = (PlanningGroupInfo('left_arm/manipulator', joints=('left_arm/openarm_left_joint1', 'left_arm/openarm_left_joint2', 'le...right_joint6', 'right_arm/openarm_right_joint7'), base='openarm_body_link0', tip='openarm_right_link7', gripper=False))
last_error = None
timeout    = 120.0

dimos/e2e_tests/test_manipulation_planning_groups.py:64: TimeoutError

dimos.e2e_tests.test_dimsim_walk_forward::test_walk_forward

Stack Traces | 199s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fbbe50770>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52cf40>
human_input = <function human_input.<locals>.send_human_input at 0x7e3fba52cfe0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e4030166930>

    @pytest.mark.self_hosted_large
    def test_walk_forward(lcm_spy, start_blueprint, human_input, dim_sim) -> None:
        start_blueprint(
            "run",
            "--disable",
            "spatial-memory",
            "--disable",
            "security-module",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        origin_x, origin_y = 1, 2
        dim_sim.set_agent_position(origin_x, origin_y)
    
        human_input("move forward 3 meter")
    
>       lcm_spy.wait_until_odom_position(origin_x + 3, origin_y, threshold=0.4, timeout=120)

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e4030166930>
human_input = <function human_input.<locals>.send_human_input at 0x7e3fba52cfe0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fbbe50770>
origin_x   = 1
origin_y   = 2
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52cf40>

dimos/e2e_tests/test_dimsim_walk_forward.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/lcm_spy.py:167: in wait_until_odom_position
    self.wait_for_message_result(
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7e3fba52dd00>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fbbe50770>
        threshold  = 0.4
        timeout    = 120
        x          = 4
        y          = 2
dimos/e2e_tests/lcm_spy.py:153: in wait_for_message_result
    wait_until(
        event      = <threading.Event at 0x7e40309c6b40: unset>
        fail_message = 'Failed to get to position x=4, y=2'
        listener   = <function LcmSpy.wait_for_message_result.<locals>.listener at 0x7e3fba52e0c0>
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7e3fba52dd00>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e3fbbe50770>
        timeout    = 120
        topic      = '/odom#geometry_msgs.PoseStamped'
        type       = <class 'dimos.msgs.geometry_msgs.PoseStamped.PoseStamped'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <bound method Event.is_set of <threading.Event at 0x7e40309c6b40: unset>>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Failed to get to position x=4, y=2

deadline   = 3219303.748119503
interval   = 0.1
message    = 'Failed to get to position x=4, y=2'
predicate  = <bound method Event.is_set of <threading.Event at 0x7e40309c6b40: unset>>
timeout    = 120

.../utils/testing/waiting.py:35: TimeoutError

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_dimsim_spatial_memory::test_go_to_the_bed

Flake rate in main: 16.11% (Passed 125 times, Failed 24 times)

Stack Traces | 105s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e40dad7a210>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52cae0>
human_input = <function human_input.<locals>.send_human_input at 0x7e3fba52ccc0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e3fbbea7200>
explore_house = <function explore_house.<locals>.explore at 0x7e3fba52d3a0>

    @pytest.mark.self_hosted_large
    def test_go_to_the_bed(lcm_spy, start_blueprint, human_input, dim_sim, explore_house) -> None:
        start_blueprint(
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
>       explore_house()

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7e3fbbea7200>
explore_house = <function explore_house.<locals>.explore at 0x7e3fba52d3a0>
human_input = <function human_input.<locals>.send_human_input at 0x7e3fba52ccc0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7e40dad7a210>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7e3fba52cae0>

dimos/e2e_tests/test_dimsim_spatial_memory.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/conftest.py:249: in explore
    direct_cmd_vel_explorer.follow_points(points)
        direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e3fd8668c50>
        points     = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e3fd8668c50>
        waypoints  = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e3fd8668c50>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7e3fd8668c50>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

go2_short spans 59.9s but the window was 180s, so two thirds of every measurement was a deployment spinning on completed streams. That is the noisiest possible thing to measure and it diluted any real change 3:1.

github-actions · 2026-08-14T19:16:13Z

Bencher Report

Project	dimos
Branch	feat/ivan/rusage-profile
Testbed	ubuntu-latest

🚨 4 Alerts

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
unitree-go2-coldstart	peak-threads threads	📈 plot 🚷 threshold 🚨 alert (🔔)	210.00 (+11.97%) Baseline: 187.56	206.31 (101.79%)
unitree-go2-coldstart	disk-write-mib MiB	📈 plot 🚷 threshold 🚨 alert (🔔)	79.36 (+127.37%) Baseline: 34.91	38.40 (206.70%)
unitree-go2-coldstart	net-rx-mib MiB x 1e3	📈 plot 🚷 threshold 🚨 alert (🔔)	8.97 x 1e3 (+33.00%) Baseline: 6.74 x 1e3	7.42 x 1e3 (120.91%)
unitree-go2-coldstart	net-tx-mib MiB	📈 plot 🚷 threshold 🚨 alert (🔔)	248.77 (+35.53%) Baseline: 183.55	201.91 (123.21%)

Click to view all benchmark results

Benchmark	cpu-core-seconds	Benchmark Result seconds (Result Δ%)	Upper Boundary seconds (Limit %)	disk-read-mib	Benchmark Result MiB (Result Δ%)	Upper Boundary MiB (Limit %)	disk-write-mib	Benchmark Result MiB (Result Δ%)	Upper Boundary MiB (Limit %)	install-size-mib	Benchmark Result MiB x 1e3 (Result Δ%)	Upper Boundary MiB x 1e3 (Limit %)	net-rx-mib	Benchmark Result MiB x 1e3 (Result Δ%)	Upper Boundary MiB x 1e3 (Limit %)	net-tx-mib	Benchmark Result MiB (Result Δ%)	Upper Boundary MiB (Limit %)	peak-memory-mib	Benchmark Result MiB x 1e3 (Result Δ%)	Upper Boundary MiB x 1e3 (Limit %)	peak-threads	Benchmark Result threads (Result Δ%)	Upper Boundary threads (Limit %)	wall-seconds	Benchmark Result seconds (Result Δ%)	Upper Boundary seconds (Limit %)
unitree-go2-coldstart	📈 view plot 🚷 view threshold	42.84 (-9.41%) Baseline: 47.28	52.01 (82.35%)	📈 view plot 🚷 view threshold	48.90	200.00 (24.45%)	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	79.36 (+127.37%) Baseline: 34.91	38.40 (206.70%)				📈 view plot 🚷 view threshold 🚨 view alert (🔔)	8.97 x 1e3 (+33.00%) Baseline: 6.74 x 1e3	7.42 x 1e3 (120.91%)	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	248.77 (+35.53%) Baseline: 183.55	201.91 (123.21%)	📈 view plot 🚷 view threshold	2.76 x 1e3 (+8.28%) Baseline: 2.55 x 1e3	2.80 x 1e3 (98.44%)	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	210.00 (+11.97%) Baseline: 187.56	206.31 (101.79%)	📈 view plot 🚷 view threshold	29.38 (-4.29%) Baseline: 30.69	33.76 (87.01%)
unitree-go2-replay	📈 view plot 🚷 view threshold	85.50 (-25.47%) Baseline: 114.72	126.20 (67.75%)	📈 view plot 🚷 view threshold	14.12	200.00 (7.06%)	📈 view plot 🚷 view threshold	0.03 (-4.12%) Baseline: 0.03	0.04 (87.16%)				📈 view plot 🚷 view threshold	132.47 x 1e3 (+0.04%) Baseline: 132.41 x 1e3	145.65 x 1e3 (90.95%)	📈 view plot 🚷 view threshold	2,550.98 (+0.02%) Baseline: 2,550.52	2,805.57 (90.93%)	📈 view plot 🚷 view threshold	2.77 x 1e3 (-1.24%) Baseline: 2.80 x 1e3	3.08 x 1e3 (89.78%)	📈 view plot 🚷 view threshold	210.00 (+0.86%) Baseline: 208.20	229.02 (91.70%)	📈 view plot 🚷 view threshold	79.49 (-58.19%) Baseline: 190.11	209.12 (38.01%)
venv										📈 view plot 🚷 view threshold	9.67 x 1e3 (+0.06%) Baseline: 9.66 x 1e3	10.63 x 1e3 (90.96%)

🐰 View full continuous benchmarking report in Bencher

A fixed window measured whatever fit in it: go2_short spans 59.9s against a 180s window, so two thirds of every sample was a deployment idling on completed streams, and a slow boot silently changed how much of the recording got processed. A --replay run now defaults its own length to the recording's, so the same messages are measured every time. An explicit --run-for still wins, which is what cold start uses.

--error-on-alert turns the 10% thresholds into a gate. Disk moves to a static 200MiB budget rather than a percentage: its baseline is fractions of a MiB, where page-cache state produced a 1597% swing between runs of identical code and a percentage means nothing.

The first gated run failed on eight alerts from a commit that changed a comment and a flag. The percentage model builds its limit from the mean of history, and that history mixed three incompatible workloads -- 180s window, 70s window, and data-bound -- so the limits described a configuration that no longer exists. New testbed starts the series from the current shape, and thresholds now need five samples before they act, so the gate arms itself once it has something real to compare against instead of firing on a single point. Disk stays on an absolute budget.

It cost an extra boot per job to measure something the replay run already carries: boot is roughly half of that number either way. Dropping it halves the profile's runner time.

leshy requested review from Dreamsorcerer, mustafab0, paul-nechifor and spomichter as code owners August 14, 2026 16:37

leshy changed the title ~~profiling: record deployment CPU time and peak memory per commit~~ profiling: record deployment CPU time and peak memory per PR Aug 14, 2026

leshy added 11 commits August 14, 2026 19:47

profiling: report peak memory in MiB

bd0837d

Bencher renders bytes as '3.06 units x 1e9'. MiB reads as a number a human can act on, and the measure now carries real units.

profiling: fall back to this branch's own baseline

39fe574

main has never profiled, so every delta would render as '--' until this lands. Falling back to the branch's own last green run exercises the comparison now, and the table names the ref it used so the two cases are never confused.

profiling: render the comment from a python template

649acd2

The bash version buried the layout in a jq program. The template is now a plain string at the top of the file, and moves over 3% are bolded so a real change stands out from runner noise.

leshy added 8 commits August 14, 2026 21:08

profiling: loop() now waits with a timeout, so assert the argument

b9f95f7

profiling: two-line blocks, clickable benchmarks, no footer

66e9569

Mem and Thr move to a second line so the first reads as time alone. The benchmark name links to its Bencher plots -- every number in the comment is one point on a trend line, and the trend is what is worth opening.

profiling: carry PATH into the system scope

82aa5a6

sudoers replaces PATH via secure_path even under sudo -E, so the payload lost uv the moment the scope switched to system. The runner did report io.stat=yes ip=yes on that run, so the switch itself was right.

dimensionalOS deleted a comment from codecov Bot Aug 14, 2026

dimensionalOS deleted a comment from github-actions Bot Aug 14, 2026

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026

Merge branch 'main' into feat/ivan/rusage-profile

f1e5581

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026

profiling: run the recording, not two extra minutes of idle

7119b2e

go2_short spans 59.9s but the window was 180s, so two thirds of every measurement was a deployment spinning on completed streams. That is the noisiest possible thing to measure and it diluted any real change 3:1.

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026

leshy added 3 commits August 14, 2026 23:43

profiling: drop the cold-start run

240c6d2

It cost an extra boot per job to measure something the replay run already carries: boot is roughly half of that number either way. Dropping it halves the profile's runner time.

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026

Update deployment-profile.yml

fb55239

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 17, 2026

Dreamsorcerer added 3 commits August 17, 2026 20:05

Add install group

a8ad9a9

Avoid failing on external PRs

ead1d60

Fixes from review round

2b8d98b

Navigation Menu

Uh oh!

profiling: record deployment CPU time and peak memory per PR - #3468

profiling: record deployment CPU time and peak memory per PR#3468
leshy wants to merge 30 commits into
mainfrom
feat/ivan/rusage-profile

leshy commented Aug 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

leshy commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deployment resource footprint vs feat/ivan/rusage-profile

Uh oh!

codecov Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 4 Tests Failed:

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

🚨 4 Alerts

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leshy commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Deployment resource footprint _{vs feat/ivan/rusage-profile}

codecov Bot commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading