Repository navigation
[None][test] Upload each server's warmup time from perf-sanity - #22732
chenfeiz0326 wants to merge 1 commit into
Conversation
Record the seconds from a server's launch until its /health first returns 200 (weight loading and the executor warmup included) and upload them with the test results: d_warmup_time for an aggregated (and ctx_only) server, and one d_<ctx|gen>_<i>_warmup_time per disaggregated worker, including DWDP ctx ranks and the gen_only_no_context gen worker. A disaggregated worker's task polls its own /health from a background thread and saves the value in the test output directory, where the benchmark task reads it at upload time. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
|
No actionable comments were generated in the recent review. WalkthroughThe perf sanity tests now record server launch-to-readiness durations for aggregated and disaggregated workers. Result uploads include available durations, and the README describes the metrics. ChangesWarmup-time metric collection
Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant WarmupTimeRecorder
participant HealthEndpoint
participant ServerProcess
participant WarmupArtifact
participant ResultUpload
WarmupTimeRecorder->>HealthEndpoint: Poll /health
HealthEndpoint->>ServerProcess: Check readiness
ServerProcess-->>HealthEndpoint: Return health response
HealthEndpoint-->>WarmupTimeRecorder: Return HTTP 200
WarmupTimeRecorder->>WarmupArtifact: Save elapsed duration
ResultUpload->>WarmupArtifact: Read available duration
WarmupArtifact-->>ResultUpload: Return saved value
Suggested reviewers: Merge Risk:
|
Dev Engineer Review
WarmupTimeRecorderpolls each server's/healthin a background thread. Verify that the thread stops cleanly when a server exits before reporting healthy, becausestop()must join it before termination.MAXIMIZE_METRICSandMINIMIZE_METRICS, have no baseline, and cannot fail a test. Match keys are unchanged.disagg-gen_onlywere not run on GPU.QA Engineer Review
tests/integration/defs/perf/test_perf_sanity.py. Documentation changed:tests/integration/defs/perf/README_test_perf_sanity.md. No test-list files changed.ctx_onlyLlama,gen_only_no_contextLlama, and a local three-node disaggregated e2e Llama configuration. All passed withpytest-threadleakenabled.disagg-gen_only. Their field names (ctx_<rank>and the gen-only path) are unverified on GPU.Per-File QA Perspective
tests/integration/defs/perf/test_perf_sanity.py: Verify that aggregated, disaggregated,ctx_only, andgen_only_no_contextrows contain the expectedd_warmup_timeor per-worker fields. Verify DWDP anddisagg-gen_onlyon GPU. List status in test-db is not available.tests/integration/defs/perf/README_test_perf_sanity.md: Documentation only. QA should confirm that the described field names match the uploaded rows.Description
Perf-sanity uploads only performance metrics. This PR also uploads each server's warmup time: the seconds from the server's launch until its
/healthfirst returns 200, so weight loading and the executor warmup are included. The benchmark client's warmup requests are not.aggr-*) andctx_onlyd_warmup_timedisagg-e2e-*,disagg-gen_only-*)d_ctx_<i>_warmup_time,d_gen_<j>_warmup_time(DWDP ctx ranks arectx_<rank>)gen_only_no_contextd_gen_0_warmup_time/healthfrom a background thread (joined before the server stops) and saves the value in the test output directory. The benchmark task reads it when it uploads.MAXIMIZE_METRICS/MINIMIZE_METRICS, so they get no baseline and cannot fail a test. Match keys are unchanged.test_visual_gen_perf_sanity.py) is unchanged.Test Coverage
Ran on GB300 (oci-aga) with the CI post-merge wheel of b963a7a, whose
test_perf_sanity.pymatches this PR's base, plus a temporary dump of the rowsupload_test_results_to_databasebuilds. Every test passed, withpytest-threadleakon.aggr-gpt_oss_120b_fp4_grace_blackwell-gpt_oss_fp4_tp1_mtp0_1k1kd_warmup_time223.7aggr-ctx_only-gb200_llama-3.1-8b-bf16_1k1k_con256_ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXLd_warmup_time155.7aggr-gen_only_no_context-gb200_llama-3.1-8b-bf16_1k1k_con256_ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXLd_gen_0_warmup_time168.1disagg-e2e-of a local copy of that Llama config withnum_ctx_servers: 2(3 nodes)d_ctx_0_warmup_time163.1,d_ctx_1_warmup_time163.1,d_gen_0_warmup_time165.1In the e2e run,
CTX_0andGEN_0read 4 s more than their own server logs span from the first line toApplication startup complete: the startup before the first log line.Not run on GPU: DWDP ctx ranks and
disagg-gen_only, which use the same worker path asdisagg-e2e.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code