Skip to content

[#22735][fix] Kimi K2 tool parser: skip a call to an undeclared function in streaming too - #22736

Open
LeizhenZhang wants to merge 1 commit into
NVIDIA:mainfrom
LeizhenZhang:fix/kimi-k2-undeclared-stream
Open

LeizhenZhang wants to merge 1 commit into
NVIDIA:mainfrom
LeizhenZhang:fix/kimi-k2-undeclared-stream

Conversation

@LeizhenZhang

@LeizhenZhang LeizhenZhang commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

With --tool_parser kimi_k2, a call to a function the request does not declare is skipped by detect_and_parse but streamed by parse_streaming_increment. A streamed response carries the call and ends with finish_reason: "tool_calls". The same output without streaming has no call and ends with finish_reason: "stop".

detect_and_parse skips such a call on purpose (continue after the "undefined function" warning), and TestKimiK2ToolParser::test_undefined_tool expects it. This PR applies the same rule to streaming. In _parse_buffered_tool_call, a call whose function is not in the request's tools is held until its <|tool_call_end|> arrives and then dropped with the same warning. It gets no tool index, so the calls after it keep consecutive indices, and nothing is added to prev_tool_call_arr or streamed_args_for_tool.

The other tool parsers on main report such a call in both paths. If you prefer that Kimi K2 report it too, the fix is to remove the continue in detect_and_parse and update test_undefined_tool. I kept the behavior the existing test documents.

Fixes #22735.

Before and after

Output Path main This PR
one call to undefined_func detect_and_parse no call no call
streaming, 1, 7 or 1000 characters per increment undefined_func streamed no call
get_weather, undefined_func, search_web detect_and_parse get_weather, search_web get_weather, search_web
streaming, every chunking of test_streamed_calls_match_whole_text_parse get_weather, undefined_func, search_web get_weather, search_web (indices 0 and 1)

Test Coverage

  • New TestKimiK2ToolParser::test_undefined_tool_is_not_streamed (three chunk sizes).
  • New case undeclared_call_between_calls in _KIMI_K2_OUTPUTS, used by TestKimiK2StreamingMatchesWholeTextParse for the calls (whole output, one character, fixed sizes, every two-way split) and for the streamed text.
  • -k KimiK2 in tests/unittest/llmapi/apps/test_tool_parsers.py: 4 failed and 41 passed with main's parser (the new cases), 45 passed with this PR.
  • The whole test_tool_parsers.py: 1160 passed and 48 failed with this PR. The 48 are the same tests that fail on main in my local setup, which runs the parsers from this checkout on the 1.4.0rc1 wheel. None of them is a Kimi K2 test.
  • pre-commit on the changed files passes.

Not tested: a live server. The parser was driven directly.

PR Checklist

  • PR description clearly explains what and why.
  • PR follows the TRT-LLM coding guidelines.
  • Test cases are provided for new code paths.
  • Documentation updated as needed (not needed).

Dev Engineer Review

  • KimiK2ToolParser._parse_buffered_tool_call now drops calls to undeclared functions in streaming. This matches the existing non-streaming behavior and test_undefined_tool.
  • The parser holds an undeclared call until <|tool_call_end|>. Its text is not emitted during streaming, and the call gets no index or streaming state.
  • Risk: if a stream ends before the end token, the buffer stays unchanged. The handling of that case was not verified.
  • The diff touches only the Kimi K2 parser and its test file, so the change has no wider scope.

QA Engineer Review

  • Test file changed: tests/unittest/llmapi/apps/test_tool_parsers.py (+22 lines).
  • Added TestKimiK2ToolParser::test_undefined_tool_is_not_streamed (chunk sizes 1, 7, 1000).
  • Added the undeclared_call_between_calls case to TestKimiK2StreamingMatchesWholeTextParse.
  • Covers: undeclared calls are not streamed, declared calls keep consecutive indices, and streaming matches whole-text parsing.
  • Test-list entries in test-db/ or qa/: not available.
  • Not tested: a live server. The author reports 48 failures in the full file, which they say also occur on main. Not verified.
  • Coverage verdict: sufficient for the Kimi K2 parser.

Per-file QA perspective

  • kimi_k2_tool_parser.py: Streamed responses no longer include undeclared function calls. finish_reason should now match non-streamed output for the same text.
  • test_tool_parsers.py: Adds regression coverage for the streaming and whole-text parity of undeclared calls.

…n in streaming too

detect_and_parse skips a call to a function the request does not declare,
as test_undefined_tool expects. Streaming reported the same call, so a
streamed response carried it with finish_reason "tool_calls" while the
non-streaming response had no call. Streaming now holds such a call until
its end token and drops it without giving it a tool index.

Signed-off-by: LeizhenZhang <zhangleizhen645@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The Kimi K2 streaming parser now discards undeclared function calls after their call-end token. Regression tests cover multiple chunk sizes and an undeclared call between declared calls.

Changes

Kimi K2 streaming parser

Layer / File(s) Summary
Skip undeclared calls in streaming
tensorrt_llm/serve/tool_parser/kimi_k2_tool_parser.py, tests/unittest/llmapi/apps/test_tool_parsers.py
The parser waits for the call-end token before discarding an undeclared call. Tests verify that the call does not appear in streamed output or parser state across chunk sizes, and add a fixture with declared calls on either side.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: Minimal · up to 0e76b

Streaming responses from Kimi K2 will now drop calls to undeclared functions, as non-streaming already does. The change is small and covered by tests, so merge risk is minimal. The unchecked PR checklist item should be resolved to clear the CI check.

Security architecture risk: Minimal · up to 0e76b

Streamed responses now omit undeclared tool calls, matching non-streaming behavior. The inspected change does not broaden access or add tool-execution authority.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • observed — The changed path affects generated tool-call output for parser-backed streams using this parser. The inspected serving consumer serializes returned calls into client deltas; it does not execute the named functions.

Trust Boundaries and Controls

  • observed — The new check constrains generated function names against request-supplied tool declarations before their first emitted name or arguments. This extends an existing non-streaming control to streaming. The serving layer also retains its independent tool_choice none suppression.

Resilience and Maintainability Implications

  • inferred — For interrupted undeclared calls, finish() does not release held tool content. Completed suppressed calls leave tool identity and argument-tracking state untouched, avoiding allocation of an output slot that subsequent declared calls would inherit.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the Kimi K2 tool parser and the change: skipping calls to undeclared functions in streaming. It matches the main change in the raw summary and the PR objectives. It also includes the i…
Description check Passed The description follows the repository template. It explains the problem and the fix, and it gives a before-and-after table. It lists test coverage with specific test names and results. It explains wh…
Linked Issues check Passed Issue #22735 requires streaming and non-streaming Kimi K2 parsing to report the same calls for the same output. Non-streaming detect_and_parse skips calls to undeclared functions (continue after t…
Out of Scope Changes check Passed The whole-PR diff against the merge base touches two files. tensorrt_llm/serve/tool_parser/kimi_k2_tool_parser.py gets a 12-line branch in _parse_buffered_tool_call that implements the undeclared-…
Full details: Title check

Explanation

The title names the Kimi K2 tool parser and the change: skipping calls to undeclared functions in streaming. It matches the main change in the raw summary and the PR objectives. It also includes the issue reference and the [fix] type, which follow the repository template.

Full details: Description check

Explanation

The description follows the repository template. It explains the problem and the fix, and it gives a before-and-after table. It lists test coverage with specific test names and results. It explains why the behavior matches the existing non-streaming test. It links the issue with 'Fixes #22735'. The author left the PR checklist unchecked for the final review item, which is expected. The template's 'Description' and 'Test Coverage' sections are filled in with substantive content. A live server was not tested, and the description states this openly.

Full details: Linked Issues check

Explanation

Issue #22735 requires streaming and non-streaming Kimi K2 parsing to report the same calls for the same output. Non-streaming detect_and_parse skips calls to undeclared functions (continue after the warning). The diff adds an undeclared-function branch to _parse_buffered_tool_call in tensorrt_llm/serve/tool_parser/kimi_k2_tool_parser.py. When current_tool_name_sent is False and the function is not in _tool_indices, the branch waits for &lt;|tool_call_end|&gt;, logs the same warning, and advances self._buffer past the call. It returns no calls. It does not change current_tool_id, prev_tool_call_arr, or streamed_args_for_tool. Declared calls keep consecutive indices. The streaming loop in parse_streaming_increment re-runs while the buffer changes, so a dropped call does not block later calls. The streamed output therefore matches the non-streaming output for the reproduction in the issue. The PR description reports new tests for three chunk sizes and a mixed-call fixture. I did not inspect the test file or run the tests, so the test results rest on the author's description. The code evidence alone supports the requirement.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff against the merge base touches two files. tensorrt_llm/serve/tool_parser/kimi_k2_tool_parser.py gets a 12-line branch in _parse_buffered_tool_call that implements the undeclared-call handling required by #22735. The test file, per the summary, adds regression tests and one fixture for the same behavior. No unrelated refactoring or API change appears in the summary or in the parser diff. The parser diff has no other hunks. The PR description also states that other behavior is unchanged.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 55c9021 and 0e76b18.

Files selected for processing: 2

  • tensorrt_llm/serve/tool_parser/kimi_k2_tool_parser.py
  • tests/unittest/llmapi/apps/test_tool_parsers.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Kimi K2 tool parser skips a call to an undeclared function in non-streaming responses but streams it

1 participant