[hubert-marek]

What changed

Qwen35Renderer (Qwen3.5 / 3.6 / 3.8, so Qwen3.8-Flash-Next too) and Qwen3VLRenderer raised NotImplementedError("Video parts are not yet supported by ...") on any video content part. They now render videos with a sibling of the image path in every place images go: user and tool content, render and bridge_to_next_turn, with process_multimodal on and off, and with add_vision_id (Video N: ).

• process_multimodal=False (the path verifiers' train client uses): the template's <|vision_start|><|video_pad|><|vision_end|> goes out as is, exactly like the single <|image_pad|> for images, and vLLM expands it from the raw video_url content part. client._content_parts already forwarded video_url. The bridge counts prior videos from the logical prompt's <|video_pad|> tokens, as it counts images.
• process_multimodal=True: the renderer decodes, samples and processes the video, then expands the <|video_pad|> as vLLM 0.32's Qwen3-VL processor does. mm_items["video"] carries pixel_values_videos and video_grid_thw, and mm_hashes and mm_placeholders are filled as they are for images.
• PlaceholderRange gets an optional is_embed: list[bool] | None = None, mirroring vLLM's field of the same name. A video's range also holds its frame timestamps and per-frame vision markers, so is_embed marks which positions are pads. _build_mm_token_type_ids honors it, so build_training_sample flags only the pads as video (2), which matches HF's mm_token_type_ids. Images keep is_embed=None and behave exactly as before.
• MULTIMODAL_MODELS lists "video" for the 13 Qwen3-VL / 3.5 / 3.6 / 3.8 checkpoints. All of them ship a video_preprocessor_config.json.

How the layout matches vLLM 0.32.0

• Frames (renderers/qwen3_vl.py _load_video / _decode_video): these mirror Qwen3VLVideoBackend.load_bytes with its default OpenCV codec. That covers the stream-buffered cv2.VideoCapture, the frame-count edit-list check, the 24 fps fallback, num = int(total / fps * 2) clamped to [4, 768], linspace(...).round() indices, the same grab/retrieve loop, and the same metadata (frames_indices, do_sample_frames = len(valid) == total). A (frames, metadata) pair, the shape vLLM's loaders return, is also accepted without decoding.
• Processing (_run_video_processor): the HF video processor runs with VideoMetadata and do_sample_frames from that metadata, as Qwen3VLMultiModalProcessor._apply_hf_processor_main and prime-rl#4390's trainer do. temporal_patch_size is 2, merge_size 2, patch_size 16.
• Timestamps (_video_timestamps): this is Qwen3VLProcessingInfo._get_video_second_idx, the mean seconds of each frame pair. When the HF processor samples the frames itself, its indices are recomputed first.
• Tokens (_video_placeholder_ids; emit_video in qwen3_vl.py and qwen35.py): this is get_video_repl. Per frame pair it emits <t seconds>, tokenized on its own, then <|vision_start|>, H*W/4 × <|video_pad|> and <|vision_end|>. The template's own <|vision_start|> / <|vision_end|> stay around the expansion, because vLLM targets only the bare <|video_pad|> with transformers ≥ 5.10 processors. The placeholder spans the frame groups, as vLLM's does.

Tests

• New tests/test_qwen_video.py:
  • test_video_prompt_matches_vllm: a clip with the measured 4 s, 512×288, 10 fps sampling renders to vLLM 0.32.0's prompt_token_ids for the real clip, all 617 tokens, hard-coded. It checks placeholder (4, 608), grid [4, 18, 32], and 576 pads.
  • test_video_expansion_follows_vllm_get_video_repl: checks the frame-group layout, is_embed and is_content on Qwen3.8-Flash-Next and Qwen3-VL-4B.
  • test_video_mm_token_type_ids_mark_only_pads: covers build_training_sample.
  • test_deferred_video_leaves_one_pad_for_vllm: covers process_multimodal=False, including bridge numbering of Video 2: .
  • test_video_timestamps_follow_vllm: offline; covers both sampling branches.
  • test_video_url_decodes_like_vllm: a synthetic mp4 written with OpenCV. It is skipped without cv2.
• tests/test_multimodal.py: the (model, modality) matrix now runs video for the Qwen-VL family. That includes token parity with processor.apply_chat_template + processor(videos=..., video_metadata=...) (HF), placeholder pads via is_embed, bridge carry-forward, tool-response videos for the Qwen3.5 family, and add_vision_id parity and bridge refusal.
• tests/test_client.py: video_url content parts, and the refusal of processed video features. tests/test_build_helpers.py: is_embed in _build_mm_token_type_ids.

Results:

• uv run --frozen pytest tests/ -m "not model": 218 passed, 2 skipped (main: 214 passed, 1 skipped)
• uv run --frozen pytest tests/test_multimodal.py tests/test_qwen_video.py tests/test_qwen38.py: 250 passed, 15 skipped (skips: the private Nemotron checkpoint, Qwen3-VL tool-response media, audio, and the OpenCV test). The Qwen subsets of test_build_helpers.py, test_bridge.py and test_is_content.py also pass: 152 passed, 3 skipped
• uv run --frozen --with opencv-python-headless pytest tests/test_qwen_video.py: 9 passed
• ruff check, ruff format --check (0.15.12): clean. ty check renderers exits 0; the only new diagnostics are two unresolved-import warnings for the optional cv2, like the existing vllm ones.
• Local checks against vLLM 0.32.0 (CPU build, transformers 5.19), not committed:
  • On the real clip, _decode_video's frames and metadata are identical to MediaConnector().fetch_video(..., video_processor="Qwen3VLVideoProcessor"). The rendered prompt equals vLLM's 617 ids with placeholder (4, 608).
  • vLLM's own Qwen3VLMultiModalProcessor.apply gives the same ids, placeholder and is_embed as the renderer for three synthetic videos with Qwen3-VL-4B, including one that vLLM keeps whole so the HF processor samples it. vLLM's pixel_values_videos are uint8 with normalization deferred to the model; normalized, they match the renderer's to 6e-8.

Not equivalent to images

• Processed video features cannot go to vLLM's /inference/v1/generate. Its features payload has no is_embed; vLLM's own PlaceholderRangeInfo has a TODO for it. A video's placeholder has timestamps and markers between the pads, so _build_qwen_vl_features raises NotImplementedError and points to process_multimodal=False. That is the path RL uses.
• Raw video sources need OpenCV, imported lazily with an install hint, as Pillow is for images. I did not add opencv-python-headless to the multimodal extra, to keep this PR from touching uv.lock. Say if you want it there.
• Processed videos are not cached. Images use image_cache_max; one processed video is tens of MB, so a cache would need its own bound.
• The eager layout assumes transformers ≥ 5.10 (bare-<|video_pad|> expansion). This matches vLLM with the transformers this repo locks (5.14.1) and prime-rl uses (5.19). Transformers 5.22 plans to make cap_pixels_per_frame the default, which changes grids for large videos in both vLLM and here.

Why

This unblocks Qwen3.8 video training. PrimeIntellect-ai/verifiers#3067 adds video_url content parts, and its train client sends media through this renderer with process_multimodal=False, which failed on any video. PrimeIntellect-ai/prime-rl#4390 is the trainer side that pairs those video_urls with vLLM's expanded placeholders.

🤖 Generated with Claude Code

Note

Add video part rendering to Qwen3-VL and Qwen3.5 renderers

• Adds VideoPart support to Qwen3VLRenderer and Qwen35Renderer with the vLLM-style expansion: timestamp text, vision markers, and video-pad tokens per frame group, plus processed features (pixel_values_videos, video_grid_thw) in multimodal data
• Raw video bytes, data URIs, HTTP(S)/file URLs, and local paths are decoded via OpenCV at a 2-fps sampling policy; each video gets a stable SHA-256 hash
• Registers video as a supported modality for the Qwen3-VL, Qwen3.5, Qwen3.6, and Qwen3.8 model families in renderers/base.py
• Deferred rendering emits a single video-pad token; bridge_to_next_turn preserves video numbering across processed and deferred turns
• Behavioral Change: PlaceholderRange gains an is_embed mask, and _build_mm_token_type_ids in renderers/base.py assigns modality type IDs only at embedded positions within masked ranges; _build_qwen_vl_features in renderers/client.py now raises NotImplementedError for video data — use process_multimodal=False for videos through the vLLM generate endpoint

^{Macroscope summarized 65a7146.}

[wiz-prime-intellect]

Wiz Scan Summary

Scanner	Findings
Vulnerabilities	-
Sensitive Data	-
Secrets	-
IaC Misconfigurations	-
SAST Findings	1
Software Management Findings	-
Total	1

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.