[tthakkal]

Problem

generate(..., process_multimodal=False) sends the raw media as content_parts and relies on the engine to expand it.
The engine must return the expanded prompt_token_ids and the mm_placeholders.
verifiers uses this mode for image prompts.

Some servers only accept a prompt that is already expanded, so they return neither field.
Dynamo's TITO /inference/v1/generate is one:

• Dynamo 1.5 fails every call with Engine response must include prompt_token_ids when process_multimodal=False.
• Dynamo 1.7.0.dev rejects the request with HTTP 400.

So no image rollout works against these servers. Text-only prompts are unaffected.

Change

New opt-in flag RENDERERS_CLIENT_MM_EXPANSION=1. It applies to process_multimodal=False calls that contain media parts:

1. Render the messages on the client with media expansion. This gives the expanded token ids and MultiModalData.
2. Check that the expanded prompt matches the caller's prompt with each placeholder widened in place. If it doesn't, raise MalformedGenerateResponseError. A renderer that is out of sync then fails loudly instead of training on mismatched tokens.
3. Send the expanded token_ids and vLLM's features payload, using the existing _build_mm_features path. No content_parts are sent.
4. Return the process_multimodal=False response shape built from the client-side expansion:
   • prompt_ids is the expanded prompt
   • renderer_prompt_ids is the caller's prompt
   • mm_placeholders is {modality: [{offset, length}]}
   • multi_modal_data is None

The flag is off by default, and the existing paths are unchanged.

It's an env var rather than a generate() argument because the caller here (verifiers' RendererClient) doesn't expose one. Happy to switch to a parameter if you prefer.

Testing

Unit test: test_generate_client_mm_expansion in tests/test_client.py checks that the request carries the expanded prompt with no content_parts, and that the result has the process_multimodal=False contract even though the server returns no prompt_token_ids. It fails without the change.

tests/test_client.py: 111 passed.

End to end: Qwen3.5-4B VLM RL with a prime-rl trainer and a Dynamo 1.7.0.dev + vLLM 0.32 rollout server, on a maze environment with image prompts.

• Without the flag, 0/6 calls succeed. With it, 6/6 calls succeed and 6/6 return logprobs.
• On 6/6 mazes, prompt tokens and placeholders are identical to plain vLLM, and pixel values are bit-identical to the HF processor.
• 50-step runs completed with the flag on.

Notes

• This is a workaround for servers without engine-side expansion. If Dynamo returns prompt_token_ids and mm_placeholders for process_multimodal=False, the flag isn't needed. The Dynamo-side fix is the TITO content_parts PR (dynamo-tito-content-parts.patch). feat(rl): support Prime-RL over native vLLM ai-dynamo/dynamo#13481 only covers pre-processed features.
• On vLLM >= 0.32, Qwen VL models normalize images on the device. Because features carries already-normalized pixel values, the server must run with mm_device_do_normalize=false, or images are normalized twice. This applies to any features request, not only this flag.
• Related, not overlapping: feat(client): add dynamo_chat transport + routed_experts to renderer generate #79, feat(client): add transport selector + dynamo_chat_nvext branch #34 and [DYNAMO] add chat nvext transport #11 (Dynamo transports), feat: report media placeholder positions in logical prompts; drop mm_token_type_id_map #211 (placeholder positions for engine-side expansion).

Note

Add opt-in client-side multimodal expansion in renderers.client.generate

• Adds a new path gated by RENDERERS_CLIENT_MM_EXPANSION=1. When process_multimodal is false and messages contain media parts, the client renders with multimodal processing enabled and sends the expanded prompt tokens plus multimodal data.
• The expanded tokens must match the caller's prompt outside the image placeholder spans. On mismatch it raises MalformedGenerateResponseError; on match it returns the original renderer_prompt_ids, the expanded prompt_ids, and the placeholder ranges, with multi_modal_data=None.
• Adds test_generate_client_mm_expansion in test_client.py covering the request token IDs, omitted content_parts, and returned placeholder range.
• Behavioral Change: the expansion path bypasses the existing max_prompt_len check, so expanded prompts are not length-limited; the path only activates when the env flag is set.

^{Macroscope summarized 5bdc116.}