Bug Description
In the current implementation of clean_conversation_for_handoff, the framework filters the conversation history by exclusively preserving text content types and discarding all other content.
While this effectively removes tool-control messages prior to routing, it inadvertently causes multi-modal content (such as uri or data attachments representing images, audio, or PDFs) to be silently dropped during agent handoffs. As a result, the receiving agent is provided with the text context but loses access to any associated multi-modal files from previous interactions.
Impact
This behavior affects multi-modal workflows within SequentialBuilder and HandoffAgentExecutor pipelines. Specifically, it prevents Vision models from accessing images uploaded in prior steps, as the multi-modal context is stripped out during the handoff process.
Proposed Solution
To resolve this, I propose updating the strict text check to a comprehensive allowlist. This approach will safely preserve necessary semantic content (such as text, data, uri, and hosted_file) while continuing to reliably filter out runtime and control payloads (such as function_call, usage, and error).
I have prepared a Pull Request that addresses this issue, complete with updated test coverage in test_handoff.py. Thank you for your time and review!
Bug Description
In the current implementation of
clean_conversation_for_handoff, the framework filters the conversation history by exclusively preservingtextcontent types and discarding all other content.While this effectively removes tool-control messages prior to routing, it inadvertently causes multi-modal content (such as
uriordataattachments representing images, audio, or PDFs) to be silently dropped during agent handoffs. As a result, the receiving agent is provided with the text context but loses access to any associated multi-modal files from previous interactions.Impact
This behavior affects multi-modal workflows within
SequentialBuilderandHandoffAgentExecutorpipelines. Specifically, it prevents Vision models from accessing images uploaded in prior steps, as the multi-modal context is stripped out during the handoff process.Proposed Solution
To resolve this, I propose updating the strict
textcheck to a comprehensive allowlist. This approach will safely preserve necessary semantic content (such astext,data,uri, andhosted_file) while continuing to reliably filter out runtime and control payloads (such asfunction_call,usage, anderror).I have prepared a Pull Request that addresses this issue, complete with updated test coverage in
test_handoff.py. Thank you for your time and review!