Skip to content

[None][test] Upload each server's warmup time from perf-sanity - #22732

Open
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/perf-sanity-warmup-time
Open

chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/perf-sanity-warmup-time

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • WarmupTimeRecorder polls each server's /health in a background thread. Verify that the thread stops cleanly when a server exits before reporting healthy, because stop() must join it before termination.
  • Missing or unreadable warmup artifacts are skipped with a warning. A broken write path can therefore leave fields absent without failing the run.
  • The new fields are informational only. They are excluded from MAXIMIZE_METRICS and MINIMIZE_METRICS, have no baseline, and cannot fail a test. Match keys are unchanged.
  • The author ran on GB300 only. DWDP context ranks and disagg-gen_only were not run on GPU.

QA Engineer Review

  • Test files changed: tests/integration/defs/perf/test_perf_sanity.py. Documentation changed: tests/integration/defs/perf/README_test_perf_sanity.md. No test-list files changed.
  • Test functions changed: not available from the supplied evidence.
  • Covered by the author's run: aggregated GPT-OSS, ctx_only Llama, gen_only_no_context Llama, and a local three-node disaggregated e2e Llama configuration. All passed with pytest-threadleak enabled.
  • Not covered: DWDP context ranks and disagg-gen_only. Their field names (ctx_<rank> and the gen-only path) are unverified on GPU.
  • Test-db or QA list membership: not available.
  • Coverage verdict: needs follow-up.

Per-File QA Perspective

  • tests/integration/defs/perf/test_perf_sanity.py: Verify that aggregated, disaggregated, ctx_only, and gen_only_no_context rows contain the expected d_warmup_time or per-worker fields. Verify DWDP and disagg-gen_only on GPU. List status in test-db is not available.
  • tests/integration/defs/perf/README_test_perf_sanity.md: Documentation only. QA should confirm that the described field names match the uploaded rows.

Description

Perf-sanity uploads only performance metrics. This PR also uploads each server's warmup time: the seconds from the server's launch until its /health first returns 200, so weight loading and the executor warmup are included. The benchmark client's warmup requests are not.

Case Fields per uploaded row
aggregated (aggr-*) and ctx_only d_warmup_time
disaggregated (disagg-e2e-*, disagg-gen_only-*) one per worker: d_ctx_<i>_warmup_time, d_gen_<j>_warmup_time (DWDP ctx ranks are ctx_<rank>)
gen_only_no_context d_gen_0_warmup_time
  • An aggregated run times its existing readiness wait.
  • A disaggregated worker's task waits for the benchmark, not for its own server, so it polls its own /health from a background thread (joined before the server stops) and saves the value in the test output directory. The benchmark task reads it when it uploads.
  • The fields are informational: they are not in MAXIMIZE_METRICS / MINIMIZE_METRICS, so they get no baseline and cannot fail a test. Match keys are unchanged.
  • VisualGen perf sanity (test_visual_gen_perf_sanity.py) is unchanged.

Test Coverage

Ran on GB300 (oci-aga) with the CI post-merge wheel of b963a7a, whose test_perf_sanity.py matches this PR's base, plus a temporary dump of the rows upload_test_results_to_database builds. Every test passed, with pytest-threadleak on.

Test id Warmup fields in the row (s)
aggr-gpt_oss_120b_fp4_grace_blackwell-gpt_oss_fp4_tp1_mtp0_1k1k d_warmup_time 223.7
aggr-ctx_only-gb200_llama-3.1-8b-bf16_1k1k_con256_ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXL d_warmup_time 155.7
aggr-gen_only_no_context-gb200_llama-3.1-8b-bf16_1k1k_con256_ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXL d_gen_0_warmup_time 168.1
disagg-e2e- of a local copy of that Llama config with num_ctx_servers: 2 (3 nodes) d_ctx_0_warmup_time 163.1, d_ctx_1_warmup_time 163.1, d_gen_0_warmup_time 165.1

In the e2e run, CTX_0 and GEN_0 read 4 s more than their own server logs span from the first line to Application startup complete: the startup before the first log line.

Not run on GPU: DWDP ctx ranks and disagg-gen_only, which use the same worker path as disagg-e2e.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

Record the seconds from a server's launch until its /health first returns
200 (weight loading and the executor warmup included) and upload them with
the test results: d_warmup_time for an aggregated (and ctx_only) server,
and one d_<ctx|gen>_<i>_warmup_time per disaggregated worker, including
DWDP ctx ranks and the gen_only_no_context gen worker.

A disaggregated worker's task polls its own /health from a background
thread and saves the value in the test output directory, where the
benchmark task reads it at upload time.

Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The perf sanity tests now record server launch-to-readiness durations for aggregated and disaggregated workers. Result uploads include available durations, and the README describes the metrics.

Changes

Warmup-time metric collection

Layer / File(s) Summary
Recorder and artifact creation
tests/integration/defs/perf/test_perf_sanity.py
A background recorder polls server health and saves the elapsed duration when the server first returns HTTP 200.
Server and worker readiness capture
tests/integration/defs/perf/test_perf_sanity.py
Aggregated and disaggregated runners record readiness durations. The gen-only worker recorder stops before process termination.
Result fields and metric documentation
tests/integration/defs/perf/test_perf_sanity.py, tests/integration/defs/perf/README_test_perf_sanity.md
Result uploads add available aggregate or per-worker durations. The README describes the fields and states that they do not receive baselines or gate regressions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant WarmupTimeRecorder
  participant HealthEndpoint
  participant ServerProcess
  participant WarmupArtifact
  participant ResultUpload
  WarmupTimeRecorder->>HealthEndpoint: Poll /health
  HealthEndpoint->>ServerProcess: Check readiness
  ServerProcess-->>HealthEndpoint: Return health response
  HealthEndpoint-->>WarmupTimeRecorder: Return HTTP 200
  WarmupTimeRecorder->>WarmupArtifact: Save elapsed duration
  ResultUpload->>WarmupArtifact: Read available duration
  WarmupArtifact-->>ResultUpload: Return saved value
Loading

Suggested reviewers: fredricz-20070104, junyixu-nv

Merge Risk: Minimal · up to 4947a

This adds informational warmup-time fields to perf-sanity result rows without affecting baselines or regression gating. The change is test-only and low risk. Two paths (DWDP context ranks and disaggregated gen_only) were not run on GPU, but they use the same worker code as the paths that were.

Security architecture risk: Minimal · up to 4947a

The timing fields are informational and use the existing performance-test workflow. The inspected change establishes no broader access or material weakening of security, failure containment, or ownership guarantees.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated added activity is confined to performance-test execution: reading worker address files, issuing health requests, writing local measurements, and including numeric values in the existing result-upload path. The inspected changes do not demonstrate expanded production authority or tenant exposure.

Trust Boundaries and Controls

  • observed — The new probe is an outbound HTTP GET, not a listening endpoint. Its address comes from files published by the harness or passed to launched workers through --report_addr. The gen-only path uses the same address file for the recorder and the existing server-address discovery flow.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title '[None][test] Upload each server's warmup time from perf-sanity' follows the repository template: a [None] ticket tag and a [test] type. It names the main change, which adds per-server warmu…
Description check Passed The description covers the change and the reasoning: it defines warmup time, lists which fields each run type uploads, explains how aggregated and disaggregated workers measure it, and states the fiel…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title '[None][test] Upload each server's warmup time from perf-sanity' follows the repository template: a [None] ticket tag and a [test] type. It names the main change, which adds per-server warmup time upload to perf-sanity results. It is concise and specific, so a teammate scanning history can tell what changed.

Full details: Description check

Explanation

The description covers the change and the reasoning: it defines warmup time, lists which fields each run type uploads, explains how aggregated and disaggregated workers measure it, and states the fields are informational and do not gate tests. The Test Coverage section lists specific test ids, the hardware used, and the results. The PR checklist is ticked. The template's '@coderabbitai summary' line is present. The Description section is written as a detailed explanation rather than a short one, which the template asks for. Overall the description is mostly complete and clear.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 97e0614 and 4947a77.

Files selected for processing: 2

  • tests/integration/defs/perf/README_test_perf_sanity.md
  • tests/integration/defs/perf/test_perf_sanity.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant