Repository navigation
feat(server): add legacy /v1/completions endpoint for the inbuilt inference server - #335
Merged
codelion merged 2 commits intoOct 10, 2026
Conversation
…erence server The built-in inference server only exposed the chat protocol at /v1/chat/completions, so OpenAI-legacy clients (text completions) could not use it. Extract the request_config parsing, approach/model resolution, client building and dispatch of proxy() into a shared _execute_completion() pipeline, and add POST /v1/completions on top of it: - prompt (str or list[str]) maps to user messages; the pipeline runs once per prompt and n still expands attempts into multiple choices - returns the OpenAI legacy shape: id cmpl-*, object text_completion, choices[].text/logprobs/finish_reason and usage; passthrough completion dicts (none approach / full-response plugins / batch mode) are mapped choice-by-choice - stream=true returns 400 in this first version (chat endpoint streams exactly as before); optillm_approach, max_tokens and other params pass through like on the chat endpoint - /v1/chat/completions responses are unchanged; conversation logging, batch mode and error mapping (500 JSON) behave as before Fixes algorithmicsuperintelligence#170
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #170
Problem
The built-in inference server only exposes the chat protocol at
/v1/chat/completions(optillm/server.py:759). OpenAI-legacy clients that speak the text completions protocol (client.completions.create, older SDKs/tools) cannot use optillm's local inference.What this PR does
Extracts the request_config parsing, approach/model resolution, client building and dispatch of
proxy()into a shared_execute_completion(data, messages, bearer_token)pipeline (CompletionOutcome/CompletionError), then addsPOST /v1/completionson top of it — no HTTP self-call, both endpoints share one dispatch path.First-version scope (requesting confirmation from @codelion):
prompt(str or list[str]) maps to a single user message; the pipeline runs once per prompt andnstill expands attempts, so choices accumulate (n completions per prompt, OpenAI legacy semantics).{"id": "cmpl-...", "object": "text_completion", "created": <unix>, "model": <dispatch model>, "choices": [{"text", "index", "logprobs": null, "finish_reason": "stop"}], "usage": {completion_tokens, completion_tokens_details.reasoning_tokens}}.stream=true→400with a pointer to/v1/chat/completions(first version; happy to add SSE in a follow-up).optillm_approach(body key orslug-model prefix),max_tokens/max_completion_tokens,temperature,stop, etc. pass through exactly like the chat endpoint; unsupported legacy-only params (echo,suffix,logprobs=true) are not implemented yet.noneapproach, full-response plugins like proxy, batch mode) are mapped choice-by-choice intotextchoices; batch mode responses are likewise converted./v1/chat/completionsresponses are byte-identical to before; conversation logging, batch-mode aggregation and error mapping (500{"error": ...}) behave as before. Auth is unchanged (the existing@app.before_requestAPI-key check covers the new route automatically).Red → green evidence
New offline tests
tests/test_completions_endpoint.py(8 tests, Flasktest_client()+ mockedchat_with_mcts/get_config, followingtests/test_approaches.py:335):server.pyonorigin/main):8 failed— the route does not exist (404s).8 passed(shape, multi-prompt, n-expansion, stream 400, prompt validation, error 500, chat shape regression guard).Regression tests (all green)
pytest tests/test_approaches.py— 6 passed (exercises the refactoredproxy()viatest_client, incl. the issue Bug: Race condition — concurrent requests corrupt MCTS parameters via shared global state #304 request-scoped mcts params test)pytest tests/test_plugins.py— 22 passedpytest tests/test_proxy_plugin.py— 9 passedpytest tests/test_reasoning_simple.py tests/test_ssl_config.py— 30 passedOPTILLM_API_KEY=optillm python tests/test_ci_quick.py— all checks passedpytest tests/test_batching.pyfails at collection withModuleNotFoundError: No module named 'test_utils'on unmodifiedorigin/maintoo (pre-existing, unrelated to this PR).python -m compilealland follow existing code style.Test plan
python -m pytest tests/test_completions_endpoint.py -vpython -m pytest tests/test_approaches.py tests/test_plugins.py tests/test_proxy_plugin.pyOPTILLM_API_KEY=optillm python tests/test_ci_quick.pyTransparency note: this PR was prepared with the assistance of an AI coding agent (ZCode); all code changes were reviewed and the full test matrix executed by the submitting author.