ERROR Error parsing response

[eric2788]

model: claude-3.7-sonnet
log:

INFO:     172.17.0.1:43416 - "POST /v1/chat/completions HTTP/1.1" 200 OK

2025-02-26T10:26:04.280148+0000 ERROR Error parsing response: {"choices":[{"finish_reason":"tool_calls","message":{"content":"I need to complete this text as a search query. Let me analyze what would be a natural continuation for this search query.","role":"assistant"}},{"finish_reason":"tool_calls","message":{"role":"assistant","tool_calls":[{"function":{"arguments":"{\"url\":\"https://www.google.com/search?q=9.11+and+9.9+which+is+bigger\"}","name":"fetch"},"id":"toolu_xxxxxxxxxxxx","type":"function"}]}}],"created":1740565563,"id":"msg_bdrk_xxxxxxxxx","model":"claude-3.7-sonnet","usage":{"prompt_tokens":1066,"total_tokens":103}}

2025-02-26T10:26:04.280808+0000 ERROR 4 validation errors for CreateChatCompletionResponse

choices.0.index

  Field required [type=missing, input_value={'finish_reason': 'tool_c...', 'role': 'assistant'}}, input_type=dict]

    For further information visit https://errors.pydantic.dev/2.10/v/missing

choices.1.index

  Field required [type=missing, input_value={'finish_reason': 'tool_c..., 'type': 'function'}]}}, input_type=dict]

    For further information visit https://errors.pydantic.dev/2.10/v/missing

object

  Field required [type=missing, input_value={'choices': [{'finish_rea...6, 'total_tokens': 103}}, input_type=dict]

    For further information visit https://errors.pydantic.dev/2.10/v/missing

usage.completion_tokens

  Field required [type=missing, input_value={'prompt_tokens': 1066, 'total_tokens': 103}, input_type=dict]

[SecretiveShell]

Can you please elaborate on what inference provider you are using?

We do not support anthropic endpoints right now.

[eric2788]

Can you please elaborate on what inference provider you are using?

We do not support anthropic endpoints right now.

I have both OpenAI and Anthropic, which are providers that support function calling and are compatible with the OpenAI schema. Could you please let me know which models currently support this?

[SecretiveShell]

Any OpenAI-compatible endpoints should work. I test officially with open upstream and vllm. There is some work on extra support in #36 but this is not yet complete. I ask what you are using specifically because the official anthropic endpoints are not openai compatible

[eric2788]

Any OpenAI-compatible endpoints should work. I test officially with open upstream and vllm. There is some work on extra support in #36 but this is not yet complete. I ask what you are using specifically because the official anthropic endpoints are not openai compatible

I am using a provider that supports function calling, and it works well in the open web UI. 🤔 Could you let me know if the response schema of this repository is different from the one used in the open web UI?

[eric2788]

this is the error when I use gpt-4o:

2025-02-26T11:45:41.492200+0000 ERROR Error parsing response: {"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"finish_reason":"stop","index":0,"message":{"content":"{ \"text\": \"Comparison of 9.11 and 9.9\" }","role":"assistant"}}],"created":1740570340,"id":"chatcmpl-B5A96WpLzQs3bAOdtnQmcdH4fg3EB","model":"gpt-4o-2024-08-06","prompt_filter_results":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"prompt_index":0}],"system_fingerprint":"fp_b705f0c291","usage":{"completion_tokens":19,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens":597,"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"total_tokens":616}}

2025-02-26T11:45:41.492937+0000 ERROR 1 validation error for CreateChatCompletionResponse

object

  Field required [type=missing, input_value={'choices': [{'content_fi...}, 'total_tokens': 616}}, input_type=dict]

    For further information visit https://errors.pydantic.dev/2.10/v/missing

INFO:     172.17.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK

INFO:     172.17.0.1:45218 - "POST /v1/chat/completions HTTP/1.1" 200 OK

ERROR:    Exception in ASGI application

Traceback (most recent call last):

  File "/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 244, in __call__

    await cancel_on_finish(lambda: self._listen_for_disconnect(receive))

  File "/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 233, in cancel_on_finish

    await coro()

  File "/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 175, in _listen_for_disconnect

    message = await receive()

              ^^^^^^^^^^^^^^^

  File "/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive

    await self.message_event.wait()

  File "/usr/local/lib/python3.12/asyncio/locks.py", line 212, in wait

    await fut

[eric2788]

this is the differences between mine and OpenAI schema (from LMOS-IO/LMOS-openai-types):

---

Common Elements

Both schemas share these basic structures:

• id
• choices array with finish_reason, index, and message
• created timestamp
• system_fingerprint
• usage with token counting

Key Differences

1. Top-level Properties

• OpenAI has additional properties:
  • object: "chat.completion"
  • service_tier: "scale"

2. Message Structure

• OpenAI's message includes:

  "message": {
    "content": "string",
    "refusal": "string",
    "tool_calls": [...],
    "audio": {...}
  }

• My message is simpler:

  "message": {
    "content": "string",
    "role": "assistant"
  }

3. Content Filtering

• My schema adds:
  • content_filter_results in choices
  • prompt_filter_results array
    Both include safety checks for hate, self_harm, sexual, and violence

4. Usage Details

• OpenAI:

  "completion_tokens_details": {
    "audio_tokens": 0,
    "reasoning_tokens": 0
  }

• My schema:

  "completion_tokens_details": {
    "accepted_prediction_tokens": 0,
    "rejected_prediction_tokens": 0,
    "audio_tokens": 0,
    "reasoning_tokens": 0
  }

5. Logprobs

• OpenAI includes detailed logprobs object in choices with token probabilities
• My schema doesn't include logprobs information

---

Interesting 🤔 Have you considered using the OpenAI Python library instead of the transformed OpenAPI one?

[eric2788]

opps, I am using other endpoint (which is also compatible to OpenAI) and it works now, let me close this issue.

but I think it would be better if we can have more compatible schema validation in future (like only check necessary one)