Skip to content

feat(server): add legacy /v1/completions endpoint for the inbuilt inference server - #335

Merged
codelion merged 2 commits into
algorithmicsuperintelligence:mainfrom
akushonkamen:feat/170-legacy-completions-endpoint
Oct 10, 2026
Merged

codelion merged 2 commits into
algorithmicsuperintelligence:mainfrom
akushonkamen:feat/170-legacy-completions-endpoint

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

Fixes #170

Problem

The built-in inference server only exposes the chat protocol at /v1/chat/completions (optillm/server.py:759). OpenAI-legacy clients that speak the text completions protocol (client.completions.create, older SDKs/tools) cannot use optillm's local inference.

What this PR does

Extracts the request_config parsing, approach/model resolution, client building and dispatch of proxy() into a shared _execute_completion(data, messages, bearer_token) pipeline (CompletionOutcome / CompletionError), then adds POST /v1/completions on top of it — no HTTP self-call, both endpoints share one dispatch path.

First-version scope (requesting confirmation from @codelion):

  • prompt (str or list[str]) maps to a single user message; the pipeline runs once per prompt and n still expands attempts, so choices accumulate (n completions per prompt, OpenAI legacy semantics).
  • Response shape: {"id": "cmpl-...", "object": "text_completion", "created": <unix>, "model": <dispatch model>, "choices": [{"text", "index", "logprobs": null, "finish_reason": "stop"}], "usage": {completion_tokens, completion_tokens_details.reasoning_tokens}}.
  • stream=true → 400 with a pointer to /v1/chat/completions (first version; happy to add SSE in a follow-up).
  • optillm_approach (body key or slug- model prefix), max_tokens/max_completion_tokens, temperature, stop, etc. pass through exactly like the chat endpoint; unsupported legacy-only params (echo, suffix, logprobs=true) are not implemented yet.
  • Passthrough completion dicts (the none approach, full-response plugins like proxy, batch mode) are mapped choice-by-choice into text choices; batch mode responses are likewise converted.
  • /v1/chat/completions responses are byte-identical to before; conversation logging, batch-mode aggregation and error mapping (500 {"error": ...}) behave as before. Auth is unchanged (the existing @app.before_request API-key check covers the new route automatically).
  • README: documents the new endpoint with a curl example in the "Local inference server" section.

Red → green evidence

New offline tests tests/test_completions_endpoint.py (8 tests, Flask test_client() + mocked chat_with_mcts/get_config, following tests/test_approaches.py:335):

  • Before the change (unmodified server.py on origin/main): 8 failed — the route does not exist (404s).
  • After the change: 8 passed (shape, multi-prompt, n-expansion, stream 400, prompt validation, error 500, chat shape regression guard).
8 failed   (pre-fix:  python -m pytest tests/test_completions_endpoint.py)
8 passed   (post-fix: python -m pytest tests/test_completions_endpoint.py)

Regression tests (all green)

  • pytest tests/test_approaches.py — 6 passed (exercises the refactored proxy() via test_client, incl. the issue Bug: Race condition — concurrent requests corrupt MCTS parameters via shared global state #304 request-scoped mcts params test)
  • pytest tests/test_plugins.py — 22 passed
  • pytest tests/test_proxy_plugin.py — 9 passed
  • pytest tests/test_reasoning_simple.py tests/test_ssl_config.py — 30 passed
  • OPTILLM_API_KEY=optillm python tests/test_ci_quick.py — all checks passed
  • Note: pytest tests/test_batching.py fails at collection with ModuleNotFoundError: No module named 'test_utils' on unmodified origin/main too (pre-existing, unrelated to this PR).
  • Lint: repo has no lint/format config (no ruff/black/flake8/mypy); changed files pass python -m compileall and follow existing code style.

Test plan

  • python -m pytest tests/test_completions_endpoint.py -v
  • python -m pytest tests/test_approaches.py tests/test_plugins.py tests/test_proxy_plugin.py
  • OPTILLM_API_KEY=optillm python tests/test_ci_quick.py

Transparency note: this PR was prepared with the assistance of an AI coding agent (ZCode); all code changes were reviewed and the full test matrix executed by the submitting author.

…erence server

The built-in inference server only exposed the chat protocol at
/v1/chat/completions, so OpenAI-legacy clients (text completions) could
not use it. Extract the request_config parsing, approach/model resolution,
client building and dispatch of proxy() into a shared _execute_completion()
pipeline, and add POST /v1/completions on top of it:

- prompt (str or list[str]) maps to user messages; the pipeline runs once
  per prompt and n still expands attempts into multiple choices
- returns the OpenAI legacy shape: id cmpl-*, object text_completion,
  choices[].text/logprobs/finish_reason and usage; passthrough completion
  dicts (none approach / full-response plugins / batch mode) are mapped
  choice-by-choice
- stream=true returns 400 in this first version (chat endpoint streams
  exactly as before); optillm_approach, max_tokens and other params pass
  through like on the chat endpoint
- /v1/chat/completions responses are unchanged; conversation logging,
  batch mode and error mapping (500 JSON) behave as before

Fixes algorithmicsuperintelligence#170
@codelion
codelion merged commit 086c678 into algorithmicsuperintelligence:main Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the legacy /completions endpoint

2 participants