feat: run wan2.2 VBench accuracy from the prebuilt image — bake scorer in + fix cold-start failures

[wu6u3tw]

Make the wan2.2 VBench accuracy scorer both shippable and runnable from the prebuilt endpoints image. Two parts:

1. Package it — bake the VBench accuracy project into the image (it lives under examples/, so the stock Dockerfile.dev never included it).
2. Fix it — three independent cold-start failures in the scoring path, each surfacing only after a full generation pass (30-50 min on 18 GB300 nodes).

With both, a full wan2.2 Offline accuracy run completed on GB300-NVL72 and scored 0.69998 over 248 samples. Supersedes #414 (packaging split), which is now closed. Split from #401 (part 2 of 2; the drain-timeout config is in the companion PR).

Packaging: bake the scorer into the image (scripts/Dockerfile.dev)

Adds a PROVISION_VBENCH build arg (default 1, consistent with the existing PROVISION_DSR1=1) that COPYs examples/09_Wan22_VideoGen_Example/accuracy to /opt/vbench_accuracy and builds its isolated .venv (own uv-managed Python 3.11; UV_PROJECT_ENVIRONMENT overridden per call so its old/heavy deps never leak into the main /opt/venv). The pyproject is patched to relax requires-python and swap decord->decord2 for wheel-only installs on both x86_64 and aarch64/GB300. Also installs libgl1/libglib2.0-0 (opencv). Build with --build-arg PROVISION_VBENCH=0 for a lean image without wan2.2 accuracy. Accuracy runs then reuse the baked venv via UV_NO_SYNC=1 (vbench_project_path=/opt/vbench_accuracy) — no host checkout.

Fix 1: stage videos as .mp4 regardless of source suffix

VBenchScorer._stage_videos kept the source extension (src.suffix or '.mp4'). VBench's load_video dispatches purely on the extension and raises bare NotImplementedError for anything but .mp4/.gif/frame dirs. trtllm-serve emits MJPEG .avi when ffmpeg is not installed server-side, so a full accuracy pass generated all 248 videos and then died at scoring. decord (libav) detects the container by content, not extension; an MJPEG-AVI symlinked under an .mp4 name decodes correctly (verified on GB300: 81 frames, 720x1280). Staged symlinks are now always named {prompt}-{idx}.mp4.

Fix 2: vbench_runner.py fails on torch >= 2.6 checkpoint loading

The wan2.2 accuracy subproject resolves torch 2.12, where torch.load defaults to weights_only=True (changed in 2.6). VBench's reference checkpoints (motion_smoothness AMT, RAFT) are full pickles and fail with UnpicklingError. The runner sets torch's official escape-hatch env var TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 (via os.environ.setdefault) at the top of the scorer subprocess — no monkeypatching of torch.load. It is read at torch.load call time and only overrides the default (callsites passing weights_only explicitly are unaffected), so the effect is scoped to this subprocess and the parent benchmark process keeps stock semantics.

Fix 3: opaque failure when wget/unzip are missing

vbench.utils.init_submodules downloads per-dimension weights via literal wget/unzip subprocesses on a cold cache. In a minimal client container this dies mid-evaluation with FileNotFoundError: 'wget', after the videos were already generated. The runner now preflights both tools and exits early with a structured error naming what is missing and how to fix it. (If you rely on the cold-cache download, provide wget/unzip or pre-stage the weights; otherwise bake/mount the weights.)

Testing

• New test_stage_videos_renames_non_mp4_sources_to_mp4.
• tests/unit/evaluation/test_scoring.py and the full unit suite pass with zero new failures vs the unmodified base commit.
• Hardware validation on GB300-NVL72: VBench scored 248/248 staged videos end to end (score 0.69998), including cold-cache weight downloads.
• Image build: docker build -f scripts/Dockerfile.dev --build-arg PROVISION_VBENCH=1 . (and PROVISION_VBENCH=0 leaves the stock image behavior for everything but the two apt packages).

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request improves VBench evaluation robustness by ensuring staged videos always use the ".mp4" extension to avoid loading errors, adding pre-flight checks for system dependencies ("wget" and "unzip"), and monkeypatching "torch.load" to support older VBench checkpoints on PyTorch 2.6+. The review feedback suggests a cleaner approach to configuring PyTorch's loading behavior by setting the "TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD" environment variable instead of manually monkeypatching "torch.load".

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@fad6ff2). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #403   +/-   ##
=======================================
  Coverage        ?   80.55%           
=======================================
  Files           ?      132           
  Lines           ?    17471           
  Branches        ?        0           
=======================================
  Hits            ?    14073           
  Misses          ?     3398           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arekay-nv]

Review Council — Multi-AI Code Review

Reviewed by Codex + Grok (cursor-cli) + Claude · depth: thorough

Posting 7 findings (2 high, 5 medium). Filtered: 3 low-severity consider items and 5 pre-existing/out-of-scope items on scoring.py/tests not touched by this diff. The two high findings were each flagged independently by all three reviewers and reproduced locally by building this PR's image.

🔴 Must fix — both defeat the self-contained --mode acc image

File:line	Issue
Dockerfile.dev:98	Baked at /opt/vbench_accuracy but scorer default path is examples/09_…/accuracy and no VBENCH_PROJECT_PATH is set → FileNotFoundError at init.
Dockerfile.dev:31	Runner hard-fails without wget/unzip; apt installs only libgl1 libglib2.0-0.

🟡 Should fix

File:line	Issue
Dockerfile.dev:105	uv sync no CUDA pin → possibly CPU-only torch on aarch64 (verify).
Dockerfile.dev:92	Comment claims UV_NO_SYNC=1 reuse, but it is set nowhere.
vbench_runner.py:98	Missing-tools guard is unconditional; aborts even with a warm cache.
Dockerfile.dev:104	Build-time uv lock regenerates rather than syncing the committed lock → drift.
vbench_runner.py:117	Exit code 2 overloaded (missing-tools vs CUDA); docstring stale.

Out of scope (pre-existing, not this diff): scoring.py partial-completeness (1597), unguarded results read (1645), prompt-filename collision (1374). The .mp4 staging change itself was reviewed and is correct.

[arekay-nv]

LGTM - thanks!