Skip to content

Resolve tool schema $refs before MLX chat template rendering#177

Open
Adi2K wants to merge 1 commit into
huggingface:mainfrom
Adi2K:fix/normalize-tool-schema-types
Open

Resolve tool schema $refs before MLX chat template rendering#177
Adi2K wants to merge 1 commit into
huggingface:mainfrom
Adi2K:fix/normalize-tool-schema-types

Conversation

@Adi2K

@Adi2K Adi2K commented Jul 21, 2026

Copy link
Copy Markdown

Any tool with a nested-object parameter currently fails every Gemma request with runtime("upper filter requires string"), before a token is generated.

GenerationSchema encodes a nested tool parameter as a bare $ref into $defs, and withResolvedRoot() resolves only the top-level root reference, so the nested part reaches the chat template as a typeless stub. For a tool search_records(filter: { q: String }):

// before — Gemma's format_parameters does `value['type'] | upper`, which throws
"properties": { "filter": { "$ref": "#/$defs/Filter" } }

// after — inlined, so the template renders and the model can target filter.q
"properties": { "filter": { "type": "object", "properties": { "q": { "type": "string" } } } }

Cloud backends accept $ref because they dereference server-side; on-device templates dereference nothing, so the MLX tool path has to resolve before rendering. convertToolToMLXSpec now runs normalizeToolSchemaTypes(resolveToolSchemaRefs(_:)). Draft-07 definitions resolve alongside $defs, since Pydantic v1 and many MCP servers emit those.

Expansion is bounded, because these schemas can come from third-party MCP servers: cycles, unfollowable pointers, and anything past the depth (64) or node (5,000) cap collapse to {"type": "string"} rather than throwing. The node cap is the load-bearing one — a definition referenced twice by one parent is a DAG rather than a cycle, so a cycle guard alone still re-inlines every shared subtree, and a 2 KB 22-level diamond expands to ~21M nodes. llama.cpp hits the same wall and sidesteps it by emitting a named grammar rule per $ref, which isn't available here since a template needs a literal schema tree.

Resolution walks every subschema slot (including additionalProperties, prefixItems, patternProperties and tuple-form items, which Pydantic emits for Dict[str, Model] and Tuple[...] fields), and a residual sweep guarantees no $ref survives anywhere else — dropping the definition table while leaving a pointer behind would be worse than not resolving at all. Slots carrying instance data rather than subschemas (const, default, enum, examples) pass through untouched, since a $ref key there is a literal value.

Well-formed schemas pass through byte-identical, and cloud backends are untouched.

Two things I'd value your opinion on:

  • Degradation is silent. A schema past the bounds is flattened with no diagnostic, which matches the existing dangling-ref behavior. I didn't add logging because the package has none today, but I'm happy to add a counter or a strict mode if you'd want one.
  • Unresolvable refs become {"type": "string"}, which asserts a type that may not be true. A permissive {} would be more honest, but templates that index value['type'] need the key present.

@Adi2K
Adi2K force-pushed the fix/normalize-tool-schema-types branch from aa87b06 to 920d933 Compare July 21, 2026 16:18
@Adi2K Adi2K changed the title Resolve tool-schema $ref before MLX chat-template rendering Resolve tool schema $refs before MLX chat template rendering Jul 21, 2026
GenerationSchema hoists named nested schemas into $defs and emits the
referencing property as a bare {"$ref": "#/$defs/<name>"} with no "type".
withResolvedRoot() resolves only the top-level root reference, so a
nested-object or union tool parameter reaches the model as a typeless stub.

Cloud backends accept this, because they dereference $ref server-side.
On-device chat templates do not: Gemma's format_parameters applies
`value['type'] | upper` to every property and throws "upper filter requires
string" on the stub's missing type, failing the whole tool-enabled request.

convertToolToMLXSpec now resolves references before the template runs:

- resolveToolSchemaRefs inlines each $ref with a resolved copy of its
  definition, so a nested object regains its {"type": "object", ...} shape.
  Draft-07 "definitions" tables resolve the same way, since Pydantic v1 and
  many MCP servers emit those. Keys beside a $ref merge onto the target.

- Every subschema slot is walked, because Dict[str, Model] emits its $ref
  under additionalProperties and Tuple[Model, int] under prefixItems. The
  tables are dropped whether or not a slot was walked, so a $ref outliving
  them reaches the template dangling: any the walk leaves collapses too.

- Expansion is bounded, since these schemas can come from third-party MCP
  servers. Cycles and pointers this resolver cannot follow collapse to
  {"type": "string"}. A definition referenced twice by one parent is a DAG
  rather than a cycle, so a path-only guard re-inlines every shared subtree:
  a 2 KB 22-level diamond expands to ~21M nodes. A node budget caps that,
  and a depth cap keeps a long chain or deep literal schema off the stack.

- normalizeToolSchemaTypes then gives any residual typeless node a
  renderable scalar type, coerces non-string "type" values, and replaces a
  "properties" or "items" that is not a schema mapping.

Well-formed schemas pass through unchanged, and cloud backends are untouched.
@Adi2K
Adi2K force-pushed the fix/normalize-tool-schema-types branch from 920d933 to b8b27ee Compare July 21, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant